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FOREWORD 


The  cost  of  training  devices  and  simulators  has  exceeded, 
in  some  cases,  the  cost  of  the  operational  equipment  that  they 
service.  The  capabilities  for  simulating  reality  are  simultane¬ 
ously  increasing  on  an  annual  basis.  The  problem  confronted  by 
the  military  is  to  determine  exactly  how  much  simulation  is 
sufficient  for  the  stated  learning  objectives.  Behavioral  and 
analytical  techniques  that  can  quickly  and  easily  project  or 
predict  how  much  simulation  and  training  is  required  are  lacking. 
At  the  same  time  information  on  the  cost-effective  use  of  training 
equipment  within  courses  of  instruction  is  sparse.  The  develop¬ 
ment  of  models,  databases,  and  techniques  addressing  these  prob¬ 
lems  provides  the  first  steps  toward  providing  integrated  behav¬ 
ioral  and  engineering  decisions  in  designing,  fielding,  and  using 
advanced  training  technology.  The  potential  effect  on  the  Army  is 
to  reduce  the  cost  of  training  equipment  while  increasing  the 
equipment's  instructional  effectiveness. 

In  response  to  these  concerns  and  problems,  the  Army  Research 
Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  and  the 
Project  Manager  for  Training  Devices  (PM  TRADE)  have  joined  ef¬ 
forts  (MOU  of  Technical  Coordination,  May  1983;  MOU  Establishing 
the  ARI  Orlando  Field  Unit,  March  1985;  Expanded  MOU,  July  1986) . 
Task  number  3.4.5,  Advanced  Technology  for  the  Design  of  Training 
Devices  and  Simulators,  provided  the  impetus  for  the  work  reported 
in  this  document.  PM  TRADE  has  maintained  partnership  in  all  as¬ 
pects  of  the  development  of  the  models,  databases,  and  analytical 
techniques.  The  final  prototype  software  was  delivered  to  ARI  and 
PM  TRADE  in  December  1988,  and  has  been  disseminated  to  interested 
parties  at  Fort  Rucker,  the  Army  Training  Support  Command,  and  the 
Systems  Training  Directorate  at  the  Training  and  Doctrine  Command. 
The  prototype  has  also  been  provided  at  their  request  to  the  Naval 
Training  Systems  Center  Human  Factors  Research  Group,  the  Air 
Force  Aeronautical  Systems  Division,  the  Air  Force  Human  Research 
Laboratory  at  Williams  AFB,  and  National  Aeronautics  and  Space 
Administration  Ames  Research  Center.  The  models  and  techniques 
developed  in  this  effort  are  expected  to  provide  the  basis  for 
useful  aids  supporting  the  integration  of  behavioral  and  engineer¬ 
ing  data,  knowledge,  and  expertise  in  training  equipment  design  in 
the  future. 
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OPTIMIZATION  OF  SIMULATION-BASED  TRAINING  SYSTEMS:  MODEL 
DESCRIPTION,  IMPLEMENTATION,  AND  EVALUATION 


EXECUTIVE  SUMMARY 


Requirement: 

The  goal  of  this  project  is  to  develop  and  demonstrate 
methods  that  help  training-device  designers  perform  tradeoff 
analyses  required  for  training-device  design.  These  methods 
should  allow  the  designer  to  determine  the  training-device  al¬ 
ternatives  that  meet  training  requirements  at  a  minimum  cost  or 
provide  the  maximum  training  f>f fectiveness  at  a  given  cost.  The 
methods  should  be  usable  and  apply  to  the  concept-formulation 
phase  of  the  training-device  development  process. 


Procedure: 

A  model  for  the  optimization  of  simulation-based  training 
systems  (OS BATS)  was  developed  using  a  systematic,  top-down  design 
procedure.  The  model  consists  of  five  tools  that  address  the  fol¬ 
lowing  problems:  (a)  determining  which  tasks  should  be  trained  by 
part-mission  or  full-mission  simulators,  and  which  should  be 
trained  on  actual  equipraent;  (b)  specifying  which  instructional 
features  are  needed  to  train  a  set  of  tasks  efficiently; 

(c)  specifying  the  levels  at  which  fidelity  should  be  provided 
along  several  fidelity  dimensions  in  order  to  meet  task  training 
requirements  and  satisfy  a  training-device  cost  limit;  (d)  deter¬ 
mining  the  group  of  training  devices  that  can  train  all  required 
tasks  at  the  minimum  cost;  and  (e)  determining  the  optimal  alloca¬ 
tion  of  training  time  to  training  devices,  given  constraints  on 
device  use.  The  tools  share  common  data  on  task  requirements, 
training  device  features,  and  costs.  A  prototype  Decision  Support 
System  (DSS)  implementing  the  OSBATS  model  was  developed  and  for- 
matively  evaluated.  The  model  was  demonstrated  on  Army  rotary¬ 
wing  aviation  tasks,  and  specifications  for  application  to  armor 
maintenance  were  developed. 


Findings: 

The  OSBATS  model  is  described  as  a  system  model  using  the 
IDEFO  (Integrated  Computer-Aided  Manufacturing  Definition)  system 
modeling  language.  The  IDEFO  model  provides  a  top-down  analysis 
of  model  components  and  relationships.  In  addition,  the  model 
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activities  are  described  for  an  example  problem  from  a  user's 
perspective.  The  example  problem  illustrates  the  way  that  the 
OSBATS  model  organizes  the  decision  process,  the  kinds  of  analyses 
that  can  be  performed,  and  the  kinds  of  output  that  are  produced. 
Finally,  the  results  of  a  formative  evaluation  and  an  analysis  of 
the  activities  required  to  apply  the  OSBATS  model  to  the  armor 
maintenance  domain  are  described. 


Use  of  Findings: 

The  OSBATS  model  may  be  used  by  engineers  responsible  for  the 
development  of  a  training-device  concept  to  perform  tradeoff 
analyses  required  to  support  the  selection  of  the  best  technical 
approach  to  a  training-device  requirement.  The  prototype  DSS 
provides  an  interactive  environment  in  which  the  engineer  may 
perform  several  kinds  of  tradeoff  analyses.  The  OSBATS  software 
includes  the  data  necessary  to  use  the  model  for  certain  limited 
problems  in  Army  rotary-wing  aviation. 
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OPTIMIZATION  OF  SIMULATION-BASED  TRAINING  SYSTEMS: 

MODEL  DESCRIPTION,  IMPLEMENTATION,  AND  EVALUATION 

INTRODUCTION 

This  report  describes  a  project  to  develop,  implement, 
and  evaluate  a  prototype  decision  support  system  (DSS)  that 
helps  training-device  designers  provide  for  the  Optimization  of 
Simulation-Based  Training  Systems  (OSBATS) .  The  goal  of  the 
OSBATS  model  is  to  specify  training  concepts,  training-device 
designs,  and  allocations  of  time  to  different  training  device 
alternatives  that  either  minimize  the  training  device  costs 
required  to  meet  training  objectives  or  maximize  the  training 
device  effectiveness  obtained  at  a  specified  cost. 

The  first  year  of  this  effort  produced  the  conceptual 
framework  for  the  OSBATS  model  (Sticha,  Blacksten,  Buede,  & 
Cross,  1986) ,  plans  for  implementation  and  evaluation  (Young, 
Luster,  Stock,  Mumaw,  &  Sticha,  1986),  and  a  review  of  the 
research  literature  relevant  to  the  goals  of  the  OSBATS  model 
(Sticha,  Blacksten,  Knerr,  Morrison,  &  Cross,  1986) .  In  the 
second  year,  our  effort  concentrated  on  revising  and  enhancing 
the  model,  developing  prototype  decision-support  software 
implementing  the  OSBATS  model  (Elder,  Gilligan,  &  Sticha,  1988) , 
and  conducting  a  preliminary  evaluation  of  software  modules 
(Sticha,  Singer,  Blacksten,  Mumaw,  &  Buede,  1987) .  Development 
and  evaluation  efforts  have  continued  during  the  final  year  of 
the  project.  This  report  describes  the  prototype  OSBATS  model, 
its  implementation,  and  the  results  of  the  formative  evaluation. 

Goals  of  the  OSBATS  Model 


A  training  system  consists  of  several  major  components, 
including  instructors,  students,  training  equipment, 
instructional  strategies,  and  tasks  to  be  trained.  The  major 
concern  of  the  OSBATS  model  is  with  the  design  and  use  of 
training  equipment,  particularly  equipment  that  simulates  the 
operation  of  part  or  all  of  the  weapon  system.  The  overall  goal 
of  the  OSBATS  model  is  to  determine  the  kinds  of  training  devices 
that  can  meet  training  requirements  at  the  minimum  cost. 

Training  requirements  are  defined  by  the  specific  tasks  that  the 
student  must  be  able  to  perform  after  training.  Within  this 
general  framework,  we  have  restricted  the  scope  of  the  OSBATS 
model  in  several  respects. 

1.  We  are  concerned  principally  with  tasks  that  can  be  trained 
by  training  devices  or  simulators.  We  are  not  concerned 
with  tasks  that  can  be  trained  adequately  in  a  classroom. 

2.  We  are  concerned  with  training  devices  that  interact  with 
the  student  dynamically  in  a  manner  that  is  analogous  to  the 
interactions  that  occur  with  actual  equipment.  This 
restriction  implies  that  we  are  not  interested  in  such 
training  media  as  movies,  videotapes,  static  representations 
of  actual  equipment,  and  other  training  aids  that  serve 
primarily  to  enhance  classroom  training.  We  are  concerned 
with  computer-based  training  (CBT)  to  the  extent  that  the 
training  involves  a  dynamic  representation  of  the  tasks 
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being  trained,  rather  than  a  static  presentation  of 
information.  There  is  a  fine  line  between  part-task 
training,  which  abstracts  critical  components  of  a  task  and 
provides  training  for  these  components  in  an  environment 
that  may  be  only  slightly  similar  to  the  actual  environment, 
and  CBT  that  merely  presents  information  to  the  student.  We 
have  not  attempted  to  define  precisely  the  boundary  between 
these  two  types  of  training. 

3.  We  are  concerned  with  institutional  training  rather  than 
unit  training.  Unit  training  involves  complexities  that 
were  judged  to  be  too  difficult  to  handle  in  an  initial 
implementation.  However,  it  should  be  possible  to 
generalize  the  model  procedures  to  apply  to  unit  training  at 
a  later  date. 

We  have  decomposed  this  overall  goal  into  three  sub-goals, 
and  have  developed  a  set  of  tools  to  meet  these  sub-goals. 

The  first  sub-goal  is  to  identify  tasks  that  are  good 
candidates  for  training  using  a  training  device.  Tasks  may  be 
candidates  for  device-based  training  for  several  reasons.  First, 
the  use  of  a  training  device  may  provide  training  at  a  lower  cost 
than  comparable  training  on  actual  eguipment.  Second,  a  training 
device  may  be  able  to  produce  special  environmental  conditions 
that  would  be  unsafe,  expensive,  or  impossible  to  produce  using 
actual  equipment.  Finally,  a  training  device  may  be  more 
efficient  by  allowing  the  student  more  repetitions  of  the  tasks 
during  the  same  training  time  than  the  actual  equipment  would,  or 
by  using  appropriate  instructional  features.  A  second  element  of 
this  sub-goal  is  to  determine  clusters  of  tasks  that  have  similar 
training-device  needs.  The  task  clusters  produced  by  this 
process  form  the  requirements  that  are  used  as  the  basis  for 
training-device  design. 

The  second  sub-goal  is  to  design  training  devices  with  a 
level  of  sophistication  and  cost  that  is  tailored  to  the 
requirements  of  the  tasks  for  which  they  are  designed.  The  major 
training  device  components  considered  in  this  problem  either 
simulate  the  equipment  and  environment  or  provide  instructional 
support  to  the  training  process.  The  simulation  components  may 
vary  with  respect  to  the  fidelity  with  which  they  represent 
corresponding  actual -equipment  components.  There  are  many 
simulation  components,  such  as  the  device's  visual  system,  motion 
system,  and  dynamic  simulatic.n  system.  The  value  of  investing  in 
different  levels  of  fidelity  for  these  components  depends  on  the 
effectiveness  of  the  components  in  reaching  the  training 
requirements.  The  device-design  process  must  determine  the 
minimum  level  of  fidelity  required  by  the  tasks  to  be  trained. 

The  effectiveness  of  instructional  features  depends  upon  the 
characteristics  of  the  tasks  to  be  trained  and  the  population  of 
students.  A  training  device  should  be  designed  with  the 
instructional  features  that  provide  the  greatest  improvement  in 
training  efficiency  given  the  tasks  to  be  trained. 
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The  third  sub-goal  is  to  determine  the  way  to  allocate 
training  resources  among  training  devices  and  actual  equipment  in 
order  to  minimize  training  cost.  In  some  situations,  it  may  be 
possible  for  a  training  device  to  provide  cost-effective  training 
on  tasks  other  than  those  for  which  it  was  designed.  In  this 
case,  the  training  device  should  be  used  for  those  tasks.  In 
other  situations,  it  may  not  be  possible  to  provide  the  required 
fidelity  to  train  a  task  at  an  acceptable  cost.  In  this  case,  it 
may  be  optimal  to  design  a  simpler  training-device  that  would 
replace  only  a  portion  of  the  training  time  on  actual  equipment. 
In  its  complete  formulation,  a  procedure  for  allocating  training 
among  training  devices  and  actual  equipment  must  consider  the 
constraints  on  the  use  of  devices  and  actual  equipment  that  come 
from  budgetary  limitations,  space  and  equipment  availability,  and 
safety  concerns. 

Some  of  the  complexity  of  training-system  design  is  caused 
by  interactions  between  the  three  sub-goals.  That  is,  although 
there  is  a  general  logical  progression  through  the  sub-problems, 
later  processes  can  provide  feedback  to  earlier  processes.  For 
example,  the  resource  allocation  process  that  addresses  the  third 
sub-goal  may  indicate  that  a  high-cost  simulator  design  leads  to 
a  lower  overall  training  cost  than  either  a  moderate-  or  a  low- 
cost  device.  This  feedback  may  lead  the  analyst  to  develop  and 
evaluate  other  high-cost  device  designs.  On  the  other  hand,  the 
resource  allocation  process  might  indicate  that  a  low-cost 
training  device  can  provide  adequate  training  effectiveness  for 
all  but  a  small  subset  of  the  tasks.  This  result  might  prompt 
the  analyst  to  design  a  new  training-device  specifically  tailored 
to  the  tasks  that  could  not  be  trained  by  the  low-cost  device. 

The  interactions  between  the  sub-goals  for  the  OSBATS  model 
imply  that  a  simple  linear  approach  to  the  problem  will  not  work 
in  some  cases.  Because  of  the  complexity  of  the  sub-goal 
interactions,  any  model  must  be  designed  to  be  used  iteratively. 
That  is,  the  results  of  individual  model  components  must  provide 
input  to  later  components  and  feedback  to  earlier  components. 

The  OSBATS  model  provides  for  iterative  application  of  its 
component  modules  with  greater  precision  at  each  application 
cycle.  The  sub-goal  interactions  also  indicate  the  need  for 
sensitivity  analyses  in  which  model  assumptions  are  varied  to 
ensure  that  the  solution  obtained  is  a  global,  rather  than  local, 
optimum. 


Model  Description 

The  OSBATS  model  was  developed  iteratively  using  a  top-down, 
system-analytic  approach.  The  overall  problem  of  training-system 
optimization  was  decomposed  into  the  sub-goals,  which  were,  in 
turn,  decomposed  into  individual  modules.  Five  software  tools 
were  developed  to  address  the  three  sub-goals.  One  tool,  the 
Simulation  Configuration  Module,  addresses  the  first  sub-goal. 

Two  tools,  the  Instructional  Feature  and  Fidelity  Optimization 
Modules,  address  the  second  sub-goal.  Two  tools,  the  Training 
Device  Selection  and  Resource  Allocation  Modules,  address  the 
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third  sub-goal.  The  function  of  the  five  tools  is  briefly 
described  below,  and  is  followed  by  an  introductory  presentation 
of  the  input  data  requirements,  the  model  processes,  and  type  of 
recommendations  generated  by  the  model. 

1.  The  Simulation  Configuration  Module  clusters  tasks  to  be 
trained  according  to  their  need  for  training  on  a  full- 
mission  simulator  (FMS) ,  one  or  more  part-mission  simulators 
(PMSs) ,  or  actual  equipment  (AE) . 

2.  The  Instructional  Feature  Selection  Module  determines  the 
relative  priority  with  which  instructional  features  should  be 
included  in  a  training  device. 

3.  The  Fidelity  Optimization  Module  determines  the  relative 
priority  of  features  that  allow  a  training  device  to 
represent  the  actual  equipment  and  operational  environment. 

4.  The  Training  Device  Selection  Module  selects  a  set  of 
training  devices  that  can  be  used  to  meet  the  training 
requirements  for  each  task  at  the  least  cost. 

5.  The  Resource  Allocation  Module  determines  the  optimal 
allocation  of  training  time  to  training  devices  and  actual 
equipment  to  meet  all  training  requirements,  considering 
constraints  on  device  procurement  and  use. 

Input  Data  Requirements 

All  methods  of  training  system  design  require  a  good  front- 
end  analysis.  The  OSBATS  model  is  no  exception  to  this  rule,  and 
requires  information  about  training  requirements,  task 
characteristics,  trainee  population  skills,  candidate  training- 
device  instructional  features,  and  fidelity  dimensions.  In 
addition,  because  the  model  is  quantitative  rather  than 
qualitative,  it  requires  numerical  estimates  for  many  of  its 
parameters.  We  do  not  anticipate  that  the  engineer  using  the 
OSBATS  model  will  be  the  principal  individual  responsible  for 
providing  input  data.  Rather,  we  see  two  principal  sources  of 
data  for  the  OSBATS  model.  First,  information  about  the  problem 
structure,  general  training-device  characteristics,  and  inference 
rules  will  be  resident  in  the  model.  The  second  class  of  data 
describes  the  specific  training  problem  addressed  by  the  OSBATS 
model.  This  information  describes  the  tasks  to  be  trained 
according  to  the  parameters  of  the  model  components.  We  do  not 
expect  that  the  user  will  have  the  subject-matter  expertise 
required  to  provide  these  data  directly.  Consequently,  we 
envision  that  ultimately  these  data  will  be  developed  through  a 
task  analysis  that  supports  the  training-device  design  process. 
Certain  inputs  are  required  of  the  user,  however.  These  inputs 
consist  of  the  critical  judgments  that  express  general  priorities 
in  training-system  design,  and  that  limit  the  scope  of  the 
problem  addressed  by  the  OSBATS  model. 
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Model  Processes 


The  overall  modeling  framework  is  based  on  methods  that 
define  the  training  strategy  that  meets  the  training  requirements 
at  the  minimum  cost.  This  framework  was  originally  described  by 
Roscoe  (1971)  and  has  been  extended  by  Povenmire  and  Roscoe 
(1973) ,  Carter  and  Trollip  (1980) ,  Bickley  (1980) ,  Cronholm 
(1985) ,  and  our  own  work  (Sticha,  Blacksten,  Buede,  &  Cross, 

1986;  Sticha,  Singer,  Blacksten,  Mumaw,  &  Buede,  1987).  In  its 
simplest  form,  the  method  compares  the  ratio  of  effectiveness  of 
two  training  alternatives  to  the  ratio  of  cost  of  the  options. 

For  example,  if  a  training  program  that  employs  one  hour  of 
training  on  a  simulator  saves  30  minutes  of  training  on  actual 
equipment,  and  the  hour  of  simulator  training  costs  as  much  as  20 
minutes  of  training  on  actual  ec[uipment,  then  the  simulator  will 
meet  the  training  requirement  at  a  lower  cost  than  actual 
equipment.  This  approach  addresses  the  tradeoff  between  the 
increased  training  time  that  is  usually  required  by  tho  use  of  a 
simulator  and  the  decreased  cost  of  that  time. 

The  goal  is  to  develop  training  device  designs  that  have  the 
fidelity  and  instructional  features  required  to  meet  the  training 
requirements  for  the  tasks  while  avoiding  extraneous  or 
inefficient  features.  We  have  applied  a  general  design 
methodology  to  the  analysis  for  training-device  design.  This 
methodology  addresses  problems  in  which  there  are  many 
alternatives  formed  by  the  factorial  combination  of  several 
dimensions.  We  have  developed  two  applications  of  this 
methodology.  The  first  application  addresses  the  instructional 
features  that  should  be  included  in  the  training  device;  the 
second  application  addresses  the  fidelity  features  that  should  be 
included. 

Model  Recommendations 


The  concept  of  operation  for  the  OSBATS  model  is  based  on 
the  iterative  use  of  the  five  model  tools  to  make  recommendations 
regarding  the  definition  of  task  clusters,  the  design  of  training 
devices,  and  the  allocation  of  training  resources  among  selected 
training  devices.  Both  the  subset  of  tools  that  are  used  and  the 
order  in  which  they  are  used  may  vary  depending  on  the 
requirements  of  the  problem  and  the  preferences  of  the  user. 
Although  the  tools  may  be  used  in  a  variety  of  orders,  the  most 
natural  order  is  to  address  the  three  sub-problems  in  the  order 
that  they  were  originally  enumerated.  Later  in  the  report,  an 
example  application  of  the  tools  in  that  order  is  presented  that 
illustrates  the  kinds  of  recommendations  that  may  be  made  by  the 
OSBATS  model . 

An  overview  of  the  OSBATS  model  is  presented  in  the  second 
section  of  the  report.  A  description  of  the  model  tools,  in 
terms  of  the  processes  used  by  the  tools  and  the  generated 
output,  is  provided  first.  That  is  followed  by  a  summary  of  the 
rulebases  used  to  identify  features,  and  a  summary  of  data 
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requirements.  The  final  portion  of  the  section  discusses  model 
the  model  assumptions. 


Model  Implementation 

The  second  major  activity  of  the  project  was  the  development 
of  software  to  embody  the  OSBATS  model.  The  implementation  of 
the  model  is  embodied  by  the  complete  model  description  provided 
in  the  third  section  of  the  report.  The  IDEFO  methodology  is 
used  to  provide  a  detailed  representation  of  the  model  as 
implemented.  The  implemented  software  provides  a  prototype  DSS 
that  can  interact  with  an  analyst  about  a  simplified  training 
system  in  Army  Aviation.  The  demonstration  example  is  based  on 
the  AH-1  Airman  Qualification  Course  (AQC) ,  and  is  described  in 
the  fifth  section  of  the  report. 

The  OSBATS  software  runs  on  an  IBM  PC/AT,  Zenith  248  or 
compatible  with  64 OK  of  memory,  and  a  high  density  storage 
device.  In  addition,  the  following  features  are  required: 

(a)  an  enhanced  graphics  adapter  (EGA)  and  color  EGA  monitor, 

(b)  80287  numeric  coprocessor,  and  (c)  a  Microsoft-compatible 
mouse. 

Individual  modules  were  delivered  for  evaluation  as  they 
were  developed.  The  evaluation  results  were  used  to  guide  the 
development  of  later  modules.  The  delivery  schedule  is  shown 
below: 

December  1986  -  Instructional  Feature  Module 

February  1987  -  Fidelity  Optimization  Module 

March  1987  -  Simulation  Configuration  Module 

April  1987  -  Training-Device  Selection  Module 

June  1987  -  Resource  Allocation  Module 

July  1987  -  Integrated  OSBATS  Prototype  (version  1.0) 

March  1988  -  Revised  OSBATS  Prototype 

October  1988  -  Final  OSBATS  Prototype  (version  1.1) 

The  final  version  of  the  software  incorporates  changes  suggested 
by  the  results  of  the  evaluation  (see  below)  of  both  the 
integrated  and  revised  versions. 

The  software  implements  all  of  the  modeling  capabilities 
described  in  the  IDEFO  presented  model  design.  The  software 
allows  the  user  to  define  task  clusters,  develop  candidate 
training-device  designs  for  individual  task  clusters,  and 
evaluate  the  cost  of  providing  training  using  these  designs  in 
various  combination  with  or  without  actual  equipment.  The 
software  does  not  provide  the  capability  to  enter  or  modify  the 
data  rec[uired  by  the  model. 

Model  Evaluation 


We  conducted  evaluations  of  the  model  and  software  at 
several  points  in  the  development  process.  A  detailed 
presentation  of  the  evaluation  efforts,  analyses,  and  conclusions 
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is  presented  in  the  fourth  section  of  the  report.  The  following 
material  provides  an  overview  of  the  evaluation  efforts,  and  how 
that  information  was  used. 

First,  we  conducted  formative  evaluations  of  the  prototype 
OSBATS  modules  as  they  were  delivered.  The  evaluation  consisted 
of  interviews  with  PM  TRADE  engineers  as  a  newly  delivered  module 
was  demonstrated.  The  primary  goal  of  the  initial  evaluation  was 
to  provide  feedback  to  the  development  process  regarding  the 
perceived  utility  of  the  model  and  the  adequacy  of  the  user 
interface. 

Our  second  evaluation  involved  a  verification  of  the  methods 
used  by  the  software  against  the  description  of  the  model.  When 
discrepancies  were  found,  they  were  examined  to  find  their 
source.  In  some  cases,  our  examination  of  the  discrepancies 
uncovered  errors  in  the  model  concepts  or  methods,  which  were 
corrected.  Discrepancies  that  were  caused  by  an  error  in  the 
implementation  of  the  model  were  communicated  to  the  programming 
staff,  so  that  they  could  be  corrected.  To  our  knowledge,  all  of 
these  errors  have  been  corrected. 

A  third  form  of  evaluation  involved  a  detailed  demonstration 
of  the  software  and  structured  interview  with  potential  model 
users  from  PM  TRADE.  The  interviews  addressed  the  presentation 
of  data  and  results,  the  clarity  of  the  model's  results,  the 
validity  of  the  model's  approach,  the  availability  of  data 
required  by  the  model,  and  the  degree  to  which  all  relevant 
information  was  included  in  the  model.  Some  of  the  results  of 
this  evaluation  have  been  used  to  guide  model  revisions.  Others 
were  used  to  guide  our  recommendations  for  future  revisions. 

A  fourth  form  of  evaluation  involved  performing  sensitivity 
analyses  on  critical  model  parameters.  In  these  analysis,  the 
values  of  input  data  were  varied,  and  the  changes  in  the  overall 
results  were  examined.  The  sensitivity  analyses  helped  us  to 
pinpoint  the  input  variables  that  are  most  critical  to  the  model 
results.  These  critical  variables  are  the  ones  that  need  to  be 
known  with  the  greatest  accuracy,  and  also  are  the  best 
candidates  for  future  research. 

Our  final  evaluation  involved  an  analysis  of  the 
requirements  for  the  application  of  the  OSBATS  model  to  a 
completely  different  domain,  namely  Armor  Turret  Maintenance. 

This  analysis  investigated  the  extent  to  which  data  variables  and 
modeling  constructs  would  need  to  be  changed  to  apply  OSBATS  to 
this  new  domain. 


Organization  of  this  Report 

The  next  or  second  section  in  the  report  contains  a  general 
overview  of  the  model  components.  The  section  describes  the  data 
they  require,  the  methods  they  use,  and  the  outputs  they  produce. 
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In  the  following  section,  we  present  a  description  of  the 
IDEFO  (Integrated  Computer-Aided  Manufacturing  Definition)  system 
and  a  formal  system  description  of  the  implemented  model  using 
the  IDEFO  system  methods.  The  IDEFO  model  provides  a  top-down 
analysis  of  model  components  and  their  relationships. 

The  fourth  section  describes  the  evaluation  activities  that 
have  been  conducted.  The  evaluations  address  the  accuracy  of  the 
model,  its  utility,  and  the  quality  of  the  user  interface. 

The  next  section  of  this  report  presents  an  example  that 
illustrates  the  application  of  the  OSBATS  model  to  a  specific 
problem  related  to  the  AH-1  AQC.  The  example  problem  illustrates 
the  kinds  of  outputs  that  are  produced  by  the  model  software  and 
the  types  of  interactions  that  are  possible.  This  example  should 
orient  the  reader  to  the  decisions  that  are  addressed  by  the 
OSBATS  model,  and  the  approaches  that  the  model  uses  to  aid  these 
decisions . 

The  following  section  describes  the  application  of  the 
OSBATS  model  to  Armor  Turret  Maintenance  training.  This 
evaluation  is  a  conceptual  analysis  that  addresses  the  generality 
of  the  OSBATS  model  to  a  variety  of  training  domains. 

The  seventh  and  final  section  of  the  report  discusses 
several  of  the  issues  that  are  relevant  to  model  development  and 
use.  It  also  summarizes  the  major  accomplishments  of  this 
project  and  describes  additional  model-development  needs. 
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MODEL  OVERVIEW 


This  section  gives  an  overview  of  the  OSBATS  model  to  prepare 
the  reader  for  the  more  detailed  description  that  is  given  in  the 
following  section.  The  descriptions  in  these  sections  focus  on  how 
the  model  works.  Thus,  we  will  describe  the  input  data 
requirements  and  model  processes  in  some  detail. 

The  section  begins  with  an  overview  of  the  model  description, 
elaborating  on  the  description  presented  in  the  introduction.  Then 
a  summary  of  each  modeling  tool  or  module,  describing  the  input 
data,  model  process,  and  model  output  is  presented.  We  then 
describe  the  two  rule  bases  that  support  the  operation  of  the 
models.  Then  we  summarize  the  input  data  requirements  for  all  of 
the  modules.  Finally,  some  of  the  simplifying  assumptions  that 
have  been  used  in  the  OSBATS  model  are  summarized. 

Model  Description 

As  stated  in  the  introduction,  the  OSBATS  model  was  developed 
iteratively  using  a  top-down,  system-analytic  approach.  The 
overall  problem  of  training-system  optimization  lead  to  the 
development  of  five  software  tools.  The  top-down  structuring 
methods  were  extended  to  the  development  of  each  tool.  First, 
general  procedures  and  analysis  strategies  were  developed.  The 
general  procedures  specify  the  kinds  of  variables  that  are  relevant 
to  the  tool  and  how  they  are  combined.  For  example,  the  general 
procedures  specify  whether  a  tool  considers  life  cycle  cost  or 
development  cost  only,  what  factors  are  considered  in  the 
determination  of  effectiveness,  and  whether  the  recommendation  of 
the  tool  is  based  on  an  effectiveness/cost  ratio  or  another 
mathematical  optimization  procedure.  Then  specific  procedures  were 
obtained  by  examining  the  current  knowledge  and  supplementing  this 
knowledge,  where  necessary,  with  reasonable  conjectures.  The 
specific  procedures  provide  the  detailed  methods  used  to  calculate 
relevant  cost  and  effectiveness  measures.  The  resulting  tools  are 
general  in  that  they  apply  to  a  wide  variety  of  training  systems, 
simulators,  and  other  training  devices.  The  input  data  used  by  the 
model  contain  information  that  is  specific  to  our  initial 
application  to  aviation  training. 

To  give  an  overview  of  the  OSBATS  model,  we  briefly  describe 
the  input  data  required  by  the  model,  the  processes  used  by  the 
model,  and  the  recommendations  produced  by  the  model. 

Input  Data  Requirements 

All  methods  of  training  system  design  require  a  good  front- 
end  analysis.  The  OSBATS  model  is  no  exception  to  this  rule,  and 
requires  information  about  training  requirements,  task 
characteristics,  trainee  population  skills,  candidate  training- 
device  instructional  features,  and  fidelity  dimensions.  In 
addition,  because  the  model  is  quantitative  rather  than 
qualitative,  it  requires  numerical  estimates  for  many  of  its 
parameters . 
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The  user  of  the  OSBATS  model  should  not  be  the  principal 
individual  responsible  for  providing  input  data.  Information  about 
the  problem  structure,  general  training-device  characteristics,  and 
associated  inference  rules  is  resident  in  the  model.  This 
information  must  be  updated  periodically  as  the  domain  of  the  model 
expands,  and  as  new  research  results  are  developed.  The  updating 
will  require  data  entry  of  some  kind,  possibly  performed  by  the 
user.  Task  data  is  input  to  describe  the  specific  training  problem 
to  be  addressed  by  the  OSBATS  model.  This  information  describes 
the  tasks  to  be  trained  according  to  the  parameters  required  by  the 
model  components.  These  data  are  traditionally  developed  through 
a  task  analysis  that  supports  the  training-device  design  process. 

There  are  inputs  required  of  the  user,  consisting  of  the 
critical  judgments  that  express  general  priorities  in  training- 
system  design,  and  that  limit  the  scope  of  the  problem  addressed 
by  the  OSBATS  model.  Direct  access  to  relevant  data  will 
eventually  have  to  be  provided  through  a  database  management 
system.  When  the  database  management  system  is  integrated  with 
the  OSBATS  model,  the  user  should  be  able  to  examine  and  modify 
much  of  the  data  used  by  the  model. 

Model  Processes 


The  overall  modeling  framework  is  based  on  methods  that  define 
the  training  strategy  that  meets  the  training  requirements  at  the 
minimum  cost.  This  framework  was  originally  described  by  Roscoe 
(1971)  and  has  been  extended  by  Povenmire  and  Roscoe  (1973)  ,  Carter 
and  Trollip  (1980),  Bickley  (1980),  Cronholm  (1985),  and  our  own 
work  (Sticha,  Blacksten,  Buede,  &  Cross,  1986;  Sticha,  Singer, 
Blacksten,  Mumaw,  &  Buede,  1987)  .  In  its  simplest  form,  the  method 
compares  the  ratio  of  effectiveness  of  two  training  alternatives 
to  the  ratio  of  cost  of  the  options.  For  example,  if  a  training 
program  that  employs  one  hour  of  training  on  a  simulator  saves  30 
minutes  of  training  on  actual  equipment,  and  the  hour  of  simulator 
training  costs  as  much  as  20  minutes  of  training  on  actual 
equipment,  then  the  simulator  will  meet  the  training  requirement 
at  a  lower  cost  than  actual  equipment.  This  approach  addresses  the 
tradeoff  between  the  increased  training  time  that  is  usually 
required  by  the  use  of  a  simulator  and  the  decreased  cost  of  that 
time. 


This  simple  formulation  of  the  cost-effectiveness  of  training 
may  be  used  to  provide  recommendations  regarding  the  optimal  mix 
of  simulator  and  actual-equipment  training  when  the  effectiveness 
of  training  on  the  simulator  is  expressed  as  a  function  of  the 
amount  of  simulator  training.  In  general,  the  effectiveness  of 
simulator  training  is  a  negatively  accelerated  function  of  the 
amount  of  training  on  the  simulator.  The  optimal  mix  of  simulator 
and  actual-equipment  training  involves  training  on  the  simulator 
until  the  marginal  cost  savings  from  reduced  actual-equipment  usage 
equals  the  marginal  cost  of  the  simulator  training.  Cronholm 
(1985)  generalized  this  finding  by  splitting  the  transfer  function 
into  three  components  representing  learning  on  the  simulator, 
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transfer  to  the  actual  equipment,  and  learning  on  the  actual 
equipment,  respectively. 

We  have  extended  the  basic  modeling  framework  in  two  ways  in 
the  OSBATS  model.  The  two  extensions  are  implemented  in  the 
Training  Device  Selection  and  Resource  Allocation  modules  of  the 
OSBATS  model.  Both  extensions  make  the  same  assumptions  about 
learning  and  transfer  processes.  The  Training  Device  Selection 
Module  makes  simplifying  assumptions  about  training  cost  so  that 
it  can  provide  an  interactive  environment  for  addressing  training- 
device  selection  alternatives.  The  Resource  Allocation  Module 
relaxes  some  of  the  assumptions  to  allocate  training  resources  to 
training  devices  considering  both  discrete  purchase  costs  and 
device  use  constraints. 

This  basic  framework,  as  we  have  extended  it,  provides  the 
overview  to  the  decision  process.  We  have  combined  these  methods, 
which  address  the  third  subgoal,  with  general  methods  addressing 
the  other  two  subgoals.  These  two  methods  include  a  preliminary 
clustering  method,  implemented  in  the  Simulation  Configuration 
Module,  and  a  training-device  design  method,  implemented  in  the 
Instructional  Feature  Selection  and  Fidelity  Optimization  Modules. 

Preliminary  clustering  method.  The  goal  of  the  preliminary 
clustering  method  used  by  the  Simulation  Configuration  Module  is 
to  review  task  requirements,  simulation  needs,  and  cost  of 
simulation  capability  in  order  to  define  clusters  of  tasks  that 
have  similar  simulation  requirements.  The  method  currently  defines 
the  following  three  classes  of  training  devices;  (a)  a  full- 
mission  simulator  (FMS)  that  simulates  many  or  all  of  the 
subsystems  of  the  actual  equipment,  (b)  one  or  more  part-mission 
simulators  (PMSs)  that  simulate  selected  equipment  subsystems,  or 
(c)  actual  equipment. 

This  evaluation  examines  device-unique  capabilities,  such  as 
training  in  unsafe  situations,  and  cost  savings  to  establish  the 
value  of  training  with  some  sort  of  training  device.  In  addition, 
the  task  requirements  for  fidelity  are  used  to  estimate  the 
development  cost  that  would  be  required  to  achieve  the  required 
fidelity  for  each  task.  Using  the  assessed  costs  and  benefits,  the 
model  sorts  the  tasks  into  three  clusters;  (a)  those  tasks  that 
should  be  trained  on  actual  equipment  because  the  benefits  of 
simulation  do  not  justify  the  expense  required  to  develop  an 
effective  training  device,  (b)  those  tasks  for  which  training  in 
a  simulated  environment  is  cost-effective  and  which  have  limited 
cue  and  response  requirements  so  that  they  require  only  a  PMS ,  and 
(c)  those  tasks  for  which  training  in  a  5imulatt.J  environment  is 
cost-ef feccive,  and  which  require  an  FMS  because  they  require  a 
high-fidelity  representation  of  the  environment  on  several 
dimensions . 

The  model  makes  its  major  recommendation  regarding  whether  a 
task  should  be  trained  on  actual  equipment  or  on  a  training  device 
by  comparing  the  required  development  cost  of  the  training  device 
to  the  potential  operating-cost  savings  brought  about  by  its  use. 
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If  the  operating-cost  savings  is  sufficient  to  recover  the 
development  cost  over  the  life  cycle  of  the  weapon  system,  a 
training  device  will  be  recommended.  Otherwise,  actual  equipment 
will  be  recommended.  The  recommendations  of  the  economic  analysis 
are  overridden,  however,  if  a  training  device  is  required  for 
safety  considerations. 

Training-device  design  methodology.  The  task  clusters 
defined  by  the  above  procedure  provide  the  requirements  for 
individual  training  devices.  The  need  at  this  point  is  to  develop 
training  device  designs  that  have  the  fidelity  and  instructional 
features  required  to  meet  the  training  requirements  for  the  tasks 
in  a  single  cluster  while  avoiding  extraneous  or  inefficient 
features.  We  have  applied  a  general  design  methodology  to  the 
analysis  for  training-device  design.  This  methodology  addresses 
problems  in  which  there  are  many  alternatives  formed  by  the 
factorial  combination  of  several  dimensions.  We  have  developed  tv o 
applications  of  this  methodology.  The  first  application  addresses 
the  instructional  features  that  should  be  included  in  the  training 
device;  the  second  application  addresses  the  fidelity  features  that 
should  be  included. 

The  OSBATS  model  views  instructional  features  as  elements  of 
training  devices  that  can  improve  training  efficiency  on  individual 
tasks.  That  is,  instructional  features  reduce  the  time  or  cost 
required  to  achieve  a  given  performance  level  on  a  training  device. 
They  do  not  affect  the  ultimate  level  of  actual-equipment 
performance  that  can  be  reached  by  using  a  training  device.  The 
number  of  tasks  aided  by  each  instructional  feature  forms  the  basis 
of  an  index  of  benefit  for  the  feature.  The  analysis  proceeds  by 
comparing  the  benefit  to  the  cost  of  incorporating  each 
instructional  feature  into  the  training  device.  The  analysis  then 
orders  the  features  according  to  the  ratio  of  benefit  to  cost. 
This  order  specifies  a  collection  of  optimal  features  as  a  function 
of  the  total  budget  for  instructional  features.  The  appropriate 
budget  for  instructional  features,  given  a  total  training-device 
budget,  is  determined  in  the  Fidelity  Optimization  Model. 

The  same  modeling  framework  is  then  used  to  address  how  much 
should  be  invested  in  the  fidelity  of  the  training  device  being 
designed.  The  model  considers  several  dimensions  of  fidelity  that 
describe  task  cue  and  response  requirements.  The  task  requirements 
on  the  fidelity  dimensions  are  compared  to  tiie  cost  of  meeting 
these  requirements  to  determine  the  dimensions  for  which  increased 
fidelity  is  justified  by  increased  training  effectiveness.  The 
output  of  this  model  is  a  set  of  possible  training-device 
configurations  applicable  to  the  task  set,  each  of  which  offers  the 
greatest  effectiveness  for  its  cost. 

The  model  makes  its  selection  based  on  the  incremental 
benefit/cost  ratio  of  the  fidelity  dimension  levels.  The  costs 
are  calculated  from  the  fidelity  levels,  and  represent  development 
costs.  The  benefits  are  based  on  the  number  of  task  fidelity 
requirements  that  are  met  for  each  fidelity  dimension  level. 
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Model  Recommendations 


The  OSBATS  model  operational  concept  is  based  on  the  iterative 
use  of  the  five  model  tools  to  make  recommendations  regarding  the 
definition  of  task  clusters,  the  design  of  training  devices,  and 
the  allocation  of  training  resources  among  selected  training 
devices.  Both  the  subset  of  tools  that  are  used  and  the  order  in 
which  they  are  used  may  vary  depending  on  the  requirements  of  the 
problem  and  the  preferences  of  the  user.  Although  the  tools  may 
be  used  in  a  variety  of  orders,  the  most  natural  order  is  to 
address  the  three  subproblems  in  the  order  that  they  were 
originally  enumerated.  An  application  of  the  tools  in  that  order 
illustrates  the  kinds  of  recommendations  that  may  be  made  by  the 
OSBATS  model . 

The  Simulation  Configuration  Module  would  be  used  first  to 
examine  the  tasks  to  be  trained  to  provides  a  preliminary 
recommendation  for  the  use  of  either  actual  equipment  or  one  or 
more  training  devices.  The  result  of  this  analysis  would  be  three 
clusters  of  tasks.  Two  of  these  clusters  define  tasks  for  which 
a  full-mission  simulator  or  part -mission  training  device  should  be 
designed . 

The  analyst  would  then  use  the  task  clusters  defined  by  the 
Simulation  Configuration  Module  as  the  basis  for  the  application 
of  the  Instructional  Feature  Selection  and  Fidelity  Optimization 
Modules.  These  two  modules  would  be  used  to  define  candidate 
training  system  designs  for  each  task  cluster.  The  output  of  the 
two  modules  is  a  range  of  options  that  vary  in  cost.  Thus,  the 
overall  results  of  the  application  of  these  modules  would  be  a 
collection  of  training  device  designs  specifying  for  each  design 
the  level  of  fidelity  on  each  fidelity  dimension  and  the  collection 
of  instructional  features  included  in  the  design.  The  analyst 
would  select  several  of  these  designs  for  further  examination. 

The  Training  Device  Selection  Module  evaluates  the  training 
device  designs  produced  in  the  previous  process.  The  analyst  would 
exercise  this  module  several  times  using  different  combinations  of 
training  devices.  For  each  combination,  the  module  would  determine 
the  number  of  tasks  that  would  be  assigned  to  each  training  device, 
the  number  of  hours  each  task  would  be  assigned  to  each  device  to 
meet  the  training  requirements  at  the  lowest  cost,  and  the  optimal 
training  cost  given  the  particular  combination  of  training  devices. 
This  model  makes  the  simplifying  assumptions  that  the  hourly  cost 
of  a  training  device  is  fixed  and  that  all  devices  are  fully 
utilized.  These  assumptions  allow  the  Training  Device  Selection 
Module  to  determine  a  solution  in  less  than  one  minute. 

When  the  analyst  was  relatively  confident  of  the  solution  of 
the  Training  Device  Selection  Module,  he  or  she  would  then 
investigate  the  solution  using  the  Resource  Allocation  Module.  It 
could  be  that  the  recommendations  of  the  Training  Device  Selection 
Module  would  require  the  procurement  of  more  training  devices  than 
would  be  feasible,  or  would  provide  some  training  on  actual 
equipment  for  tasks  in  which  such  training  violated  safety 
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regulations.  The  Resource  Allocation  Module  allows  the  analyst  to 
impose  constraints  such  as  these  on  the  training  system  and  examine 
the  resulting  optimal  solution.  The  Resource  Allocation  Module 
also  relaxes  the  simplifying  assumptions  that  were  used  by  the 
Training  Device  Selection  Module  to  estimate  training  device  cost, 
leading  to  a  more  accurate  cost  function.  As  a  result  of  its 
increased  generality,  the  Resource  Allocation  Module  takes  several 
minutes  to  reach  a  solution,  much  longer  than  the  Training  Device 
Selection  Module. 

At  many  points  in  the  analysis  process  the  analyst  has  the 
option  of  returning  to  modules  that  were  used  previously  to  refine 
the  analysis,  change  assumptions,  or  choose  different  solutions. 
For  example,  the  analyst  might  change  the  definition  of  the  task 
clusters  based  on  the  results  of  Training  Device  Selection  Module, 
or  may  use  those  results  to  select  different  candidate  device 
designs  for  evaluation. 

Summary  of  Modeling  Tools 

This  section  presents  a  brief  discussion  of  each  of  the  tools, 
including  descriptions  of  the  input  requirements,  the  outputs,  and 
an  outline  of  its  method. 

Simulation  Configuration  Module 

The  tool  for  simulation  configuration  examines  task 
characteristics  to  determine  both  the  need  for  and  the 
cost-effectiveness  of  training  on  general  classes  of  training 
devices  as  compared  to  training  on  actual  equipment.  Three  classes 
of  training  devices  are  considered  by  the  tool: 

1.  A  Full-Mission  Simulator  (FMS) ,  which  simulates  many  or  all  of 
the  subsystems  of  the  actual  equipment, 

2.  One  or  more  Part-Mission  Simulators  (PMSs) ,  which  simulate 
selected  equipment  subsystems,  or 

3.  Actual  equipment. 

This  evaluation  examines  device-unique  capabilities,  such  as 
training  in  unsafe  situations  and  cost  savings,  to  establish  the 
value  of  training  with  some  sort  of  training  device.  In  addition, 
the  task  cue  and  response  requirements,  which  are  derived  using  set 
of  rules  that  associate  activities  with  requirements,  are  used  to 
estimate  the  development  cost  required  to  achieve  the  necessary 
fidelity.  Using  the  assessed  costs  and  benefits,  the  model  sorts 
the  tasks  into  three  clusters; 

1.  Those  tasks  that  should  be  trained  on  actual  equipment  because 
the  benefits  of  simulation  do  not  justify  the  expense  reguired 
to  develop  an  effective  training  device. 


14 


2.  Those  tasks  for  which  training  in  a  simulated  environment  is 
cost-effective,  and  which  have  limited  cue  and  response 
requirements  so  that  they  require  only  a  PMS. 

3.  Those  tasks  for  which  training  in  a  simulated  environment  is 
cost-effective,  and  which  require  an  FMS,  because  they  require 
a  high-fidelity  representation  of  the  environment  on  several 
dimensions. 

Input  requirements.  The  primary  inputs  of  the  Simulation 
Configuration  Module  consist  of  ratings  of  tasks  regarding  safety 
concerns,  necessary  special  performance  conditions,  and 
requirements  for  cue  and  response  fidelity.  The  following  types 
of  inputs  are  required: 

1.  Task  learning  points.  Numerical  assessments  of  entry-level 
performance  and  the  performance  standard  for  each  task. 

2.  Task  simulation  evaluation  factors.  Ratings  regarding  safety 
concerns,  special  performance  conditions,  and  training 
effectiveness  considerations.  These  factors  are  combined  to 
produce  an  index  of  simulation  indicators. 

3.  Task  information  processing  requirements  and  activity  data. 
These  data  are  used  by  the  fidelity  rules  to  derive  the  task 
cue  and  response  requirements,  which,  in  turn  are  used  to 
estimate  the  cost  of  developing  a  training  device  that  can 
provide  training  to  the  performance  standard.  This  cost 
measure  provides  the  second  major  component  of  the  model. 

4.  Task  training  hours  and  costs.  The  time  that  would  be  required 
to  train  each  task  on  actual  equipment  to  the  performance 
standard.  For  tasks  in  which  training  on  actual  equipment  is 
impossible,  the  training  time  is  an  estimate  of  the  training 
time  required  on  a  hypothetical  training  device  with  near¬ 
perfect  fidelity  and  without  any  instructional  features. 

5.  Training-device  costs.  These  measures  describe  the 

development,  fixed  operating,  and  variable  operating  costs  of 
the  training  device  templates  and  actual  equipment.  Also 
included  in  these  data  are  estimates  of  maximum  annual 
utilization  and  the  length  of  the  device  life  cycle. 

6.  Training-device  cue  and  response  capabilities  and  instructional 
features.  These  data  describe  the  cue  and  response 
capabilities  and  instructional  features  present  in  currently 
developed  or  hypothetical  training  devices.  One  of  these 
device  descriptions  is  used  as  a  baseline  training  device  to 
estimate  the  operating-cost  savings  derived  from  simulation. 

7.  Fidelity  dimension  cost  data.  These  data  are  used  to  estimate 
the  development  cost  of  candidate  training  devices  from  the  cue 
and  response  requirements. 
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8.  Fidelity  Rules.  These  rules  derive  task  cue  and  response 

requirements  from  task  activity  data  and  information  processing 
requirements . 

9.  Training-system  data.  These  data  include  user  judgments  of 
importance  and  criteria  used  by  the  tool  to  make 
recommendations . 

Module  output.  This  module  recommends  which  tasks  should  be 
trained  on  a  full-mission  simulator,  part-mission  simulator,  or 
actual  equipment.  The  assignment  of  tasks  to  clusters  is  used  by 
the  Instructional  Feature  Selection  and  Fidelity  Optimization 
Models. 

Method.  The  module  evaluates  simulation  options  for  each  task 
by  comparing  the  benefits  of  simulation  due  to  improved  safety  or 
operating-cost  savings  to  the  cost  required  to  develop  task- 
specific  effective  simulation  capability.  For  tasks  in  which  the 
benefit  of  simulation  is  high,  or  for  which  simulation  is  required 
for  safety  concerns,  a  high-cost  simulator  may  be  justified.  For 
tasks  in  which  the  benefit  from  simulation  is  moderate,  a  simulator 
would  be  justified  only  if  it  were  less  expensive  to  develop. 

The  benefits  of  simulation  are  assessed  directly  from  ratings 
on  safety  concerns,  special  performance  conditions,  and  training 
effectiveness  factors.  Operating-cost  savings  is  determined  by 
comparing  the  cost  of  training  on  actual  equipment  to  the  projected 
cost  for  an  optimal  mix  of  training  on  both  actual  equipment  and 
a  prototype  training  device.  The  cue  and  response  requirements  for 
each  task  are  used  to  estimate  the  cost  to  develop  a  training 
device  with  the  required  capability  for  that  task. 

The  model  makes  its  major  recommendation  regarding  whether  a 
task  should  be  trained  on  actual  equipment  or  some  kind  of  training 
device  by  comparing  the  required  development  cost  to  the  potential 
operating  cost  savings.  If  the  operating  cost  savings  is 
sufficient  to  recover  the  development  cost  over  the  life  cycle  of 
the  weapon  system,  a  training  device  is  recommended.  Otherwise, 
actual  equipment  is  recommended.  The  recommendations  of  the 
economic  analysis  is  overridden,  however,  if  a  training  device  is 
required  for  safety  considerations. 
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Instructional-Feature  Selection  Module 


Instructional  features  are  elements  of  training  devices  that 
can  improve  training  efficiency  on  individual  tasks.  That  is, 
instructional  features  reduce  the  time  or  cost  required  to  achieve 
a  given  performance  level  on  a  training  device.  They  do  not  affect 
the  ultimate  level  of  actual-equipment  performance  that  can  be 
reached  using  a  training  device.  The  analysis  proceeds  by  deriving 
a  benefit  measure  for  each  instructional  feature,  and  comparing  the 
benefit  to  the  cost  of  incorporating  each  instructional  feature 
into  the  training  device.  The  number  of  tasks  aided  by  each 
instructional  feature  forms  the  basis  of  an  index  of  benefit  for 
the  feature.  The  cost  of  the  feature  is  assessed  directly.  The 
analysis  then  orders  the  features  according  to  ratio  of  benefit  to 
cost.  This  order  specifies  a  collection  of  optimal  features  as  a 
function  of  the  total  budget  for  instructional  features.  The 
appropriate  budget  for  instructional  features,  given  a  total 
training-device  budget,  is  determined  in  the  Fidelity  Optimization 
Module. 

Input  requirements.  Module  inputs  include  information  about 
both  the  tasks  assigned  for  training  on  the  training  device  and 
the  candidate  instructional  features.  The  types  of  input 
requirements  are: 

1.  Tasks  cluster  assigned  to  the  training  device.  The  list  of 
tasks  that  will  be  used  as  the  basis  of  the  assessment  of 
instructional-feature  benefit  could  be  based  on  the 
recommendations  of  the  Simulation  Configuration  Module  or  the 
Training-Device  Selection  Module,  or  tasks  could  be  selected 
by  the  user. 

2.  Task  learning  points  and  task  information  processing 
requirements.  These  data  provide  the  information  used  by  the 
instructional  feature  rules  listed  below. 

3.  Task  training  hours  and  costs.  The  time  that  would  be  required 
to  train  each  task  on  actual  equipment  to  the  performance 
standard.  For  tasks  in  which  training  on  actual  equipment  is 
impossible,  the  training  time  is  an  estimate  of  the  training 
time  required  on  a  hypothetical  training  device  with  near¬ 
perfect  fidelity  and  without  any  instructional  features.  The 
costs  are  normalized  to  produce  task  weights  used  in  the 
analysis. 

4.  Instructional  feature  rules.  These  rules  associate  task 
characteristics  with  instructional  features.  For  example,  a 
rule  might  associate  the  tutorial  feature  with  tasks  that  are 
in  early  phases  of  training  (entry  performance  level  is  low) 
and  are  difficult. 

5.  Instructional-feature  cost.  This  factor  assesses  the 
procurement  cost  associated  with  each  instructional  feature. 
The  Instructional  Feature  Module  requires  only  ratio-scale 
costs  for  the  analysis.  However,  the  actual  range  of  costs  is 


17 


required  to  incorporate  the  results  of  the  Instructional 
Features  Module  into  the  Fidelity  Optimization  Module. 

6.  Instructional  feature  weight.  This  factor  is  a  task- 
independent  measure  of  instructional-feature  value  that 
estimates  the  probability  that  an  instructional  feature  would 
be  used.  This  information  would  be  based  on  research  such  as 
that  of  Polzella  (1983)  and  would  be  updated  as  new  research 
results  were  obtained. 

Module  output.  The  major  output  of  this  module  is  the  optimal 
ordering  in  which  instructional  features  should  be  included  in  a 
training  device  as  a  function  of  the  budget  for  instructional 
features.  This  list  may  serve  as  an  input  to  the  Fidelity 
Optimization  Model,  or  it  may  be  used  directly  to  identify 
particularly  useful  instructional  features  for  the  selected  task 
cluster. 

Method .  The  analysis  proceeds  in  three  steps.  In  the  first 
step,  the  user  selecti^  the  candidate  features  that  will  be 
evaluated  in  the  analysis  and  the  cluster  of  tasks  that  will  be 
used  as  the  basis  of  the  evaluation.  The  task  cluster  may  come 
from  the  recommendations  of  the  Simulation  Configuration  Module  or 
the  Training-Device  Selection  Module,  or  may  be  selected  directly 
by  the  user.  In  the  second  step,  the  benefits  of  the  instructional 
features  are  determined.  The  instructional-feature  rules  are 
applied  to  determine  the  features  that  are  appropriate  for  each 
task.  The  appropriateness  assessments  are  combined  across  tasks 
with  the  task  and  instructional-feature  weights  to  produce  a  task- 
specific  assessment  of  instructional-feature  benefit.  Thus,  the 
benefit  of  an  instructional  feature  is  the  weighted  proportion  of 
tasks  for  which  the  feature  is  appropriate.  In  the  third  step,  the 
optimal  list  of  instructional  features  is  calculated.  The  benefit 
of  each  instructional  feature  is  divided  by  its  assessed  cost,  and 
the  features  are  ordered  according  to  the  generated  benefit-to-cost 
ratio.  One  or  more  user-selected,  optimally  ordered  packages  of 
instructional  features  may  serve  as  input  to  the  Fidelity 
Optimization  Module. 

Fidelity  Optimization  Module 

This  module  addresses  the  problem  of  how  much  should  be 
invested  in  the  sophistication  of  a  training  device  that  allows  it 
to  represent  the  task-performance  environment  with  high  fidelity. 
The  module  considers  several  dimensions  of  fidelity,  comparing  task 
cue  and  response  requirements  for  the  fidelity  dimensions  with  the 
cost  of  meeting  these  requirements  to  determine  the  dimensions  for 
which  increased  fidelity  is  justified  by  increased  training 
effectiveness.  The  output  of  this  module  is  a  set  of  possible 
training-device  configurations  applicable  to  the  task  set,  each  of 
which  offers  the  greatest  effectiveness  for  its  cost. 

Input  requirements.  The  required  input  specifies  fidelity 
dimensions  and  values,  task  cue  and  response  requirements,  and 
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output  from  the  Instructional-Feature  Selection  Module,  if 
available.  Specific  input  requirements  are  enumerated  below. 

1.  Fidelity  dimension  data.  These  data  describe  the  range  of 

options  available  for  each  fidelity  dimension  evaluated  by  the 
module.  The  variables  include  measures  of  technical 

performance  and  values  used  to  estimate  the  development  cost 
for  each  level  of  each  fidelity  dimension. 

2.  Task  information  processing  requirements  and  activity  data. 
These  data  are  used  by  the  fidelity  rules  to  derive  the  task 
cue  and  response  requirements.  The  task  cue  and  response 
requirements  describe  the  performance  requirements  for  task 
cues  and  responses  along  each  of  the  fidelity  dimensions. 

3.  The  results  of  the  Instructional-Feature  Selection  Module,  if 
they  are  available,  may  be  incorporated  in  the  module  at  the 
user's  option. 

4.  Task  training  hours  and  costs.  The  time  that  would  be  required 
to  train  each  task  on  actual  equipment  to  the  performance 
standard.  For  tasks  in  which  training  on  actual  equipment  is 
impossible,  the  training  time  is  an  estimate  of  the  training 
time  required  on  a  hypothetical  training  device  with  near¬ 
perfect  fidelity  and  without  any  instructional  features.  The 
costs  are  normalized  to  produce  task  weights  used  in  the 
analysis . 

5.  Fidelity  Rules.  These  rules  derive  task  cue  and  response 
requirements  from  task  activity  and  information  processing 
requirement  data. 

Module  output.  The  output  of  this  module  is  a  range  of 
training-device  designs.  Each  design  specifies  a  single  level  of 
sophistication  on  each  fidelity  dimension  that  is  optimal  in  the 
sense  that  there  are  no  other  designs  that  are  both  lower  in  cost 
and  greater  in  effectiveness.  Once  the  set  of  optimal  designs  has 
been  determined,  a  specific  optimal  design  can  be  defined  for 
user-specified  cost  or  benefit  levels. 

Method.  In  the  first  step,  the  user  selects  the  candidate 
fidelity  dimensions  and  levels  that  will  be  evaluated  in  the 
model.  In  the  second  step,  the  cost  and  benefit  of  those  fidelity 
levels  are  determined.  The  costs  are  calculated  by  the  same 
estimating  functions  that  are  used  in  the  Simulation  Configuration 
Module,  and  represent  development  costs.  The  benefits  are  based 
on  the  number  of  tasks  for  which  each  level  of  the  fidelity 
dimensions  would  be  adequate,  calculated  by  comparing  the  technical 
performance  of  each  fidelity  level  with  the  cue  and  response 
requirements  of  the  tasks  for  the  appropriate  fidelity  dimensions. 
The  resulting  benefit  measures  are  comparable  across  fidelity 
dimensions.  In  the  third  step,  the  optimal  training  device  designs 
are  determined  based  on  the  incremental  benefit-to-cost  ratios  of 
the  options.  These  designs  may  then  be  used  by  the  Training- 
Device  Selection  or  Resource  Allocation  Modules. 
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Training-Device  Selection  Module 


This  tool  applies  cost-effectiveness  analysis  to  select  the 
training  devices  that  meet  training  requirements  for  all  t-sks  at 
the  minimum  cost.  The  model  considers  (a)  the  fidelity  of  each 
training  device  compared  to  task  cue  and  response  requirements, 
(b)  the  instructional  features  present  in  each  training  device, 
and  (c)  the  level  of  training  conducted  on  each  training  device. 
The  model  recommends  the  optimal  assignment  of  training  devices  to 
tasks,  including  (a)  the  devices  that  should  be  used  to  train  each 
task,  (b)  the  level  of  training  at  which  each  device  should  be 
employed,  (c)  the  projected  life-cycle  cost  of  training,  and  (d) 
the  level  of  performance  expected  to  result  from  training  for  each 
selected  device. 

Input  requirements.  The  major  inputs  to  the  model  describe 
training  requirements,  device  capabilities,  and  costs.  The 
following  types  of  data  are  required: 

1.  Task  learning  points.  These  data  are  numerical  assessments  of 
entry-level  performance  and  the  performance  standard  for  each 
task. 

2.  Task  activity  data.  These  data  are  used  by  the  fidelity  rules 
to  derive  the  task  cue  and  response  requirements.  The  task  cue 
and  response  requirements  describe  the  performance  requirements 
for  task  cues  and  responses  along  each  of  the  fidelity 
dimensions. 

3.  Task  training  hours  and  costs.  These  data  estimate  the  time 
that  would  be  required  to  train  each  task  on  actual  equipment 
to  the  performance  standard.  For  tasks  in  which  training  on 
actual  equipment  is  impossible,  the  assessed  training  time 
would  represent  the  training  time  required  on  a  hypothetical 
training  device  with  near-perfect  fidelity  and  without  any 
instructional  features. 

4.  Task  information  processing  characteristics.  These  data  are 
used  to  estimate  the  effectiveness  of  the  instructional 
features  possessed  by  candidate  training  devices. 

5.  Training-device  costs.  These  measures  describe  the 
development,  fixed  operating,  and  variable  operating  costs  of 
the  training  device  templates  and  actual  equipment.  Also 
included  in  these  data  are  estimates  of  maximum  annual 
utilization  and  the  length  of  the  device  life  cycle. 

6.  Training-de^'ice  cue  and  response  capabilities  and  instructional 
features.  These  data  describe  the  fidelity  and  instructional 
features  of  both  hypothetical  training  devices  and  device 
designs  that  were  produced  by  the  Fidelity  Optimization  Module. 
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7.  Fidelity  Rules.  These  rules  derive  task  cue  and  response 
requirements  from  task  activity  data  and  information  processing 
requirements . 

8.  Instructional  feature  rules.  These  rules  associate 
characteristics  of  the  tasks  with  instructional  features. 

9.  Training-system  data.  These  data  include  assumptions  regarding 
the  annual  student  throughput,  the  maximum  effect  of 
instructional  features  of  efficiency,  and  the  effect  of 
simulation  on  setup  time. 

Module  output.  The  outputs  of  the  module  include  the  optimal 
selection  of  training  devices  for  each  task,  an  estimate  of  the 
training  criterion  on  each  device  for  each  task,  and  estimates  of 
the  overall  life-cycle  cost  and  training  effectiveness  of  the 
optimal  selection  of  training  devices. 

Method.  In  the  first  step  of  the  analysis,  cost  and  training 
effectiveness  are  calculated  for  each  task  to  be  trained,  and  the 
devices  that  meet  the  training  requirements  at  the  least  cost  are 
determined.  Device  effectiveness  is  measured  by  the  expected 
increase  in  operational  performance  resulting  from  training  on  the 
device.  Expected  operational  performance  considers  both  the 
training  criterion  on  the  training  device  (expressed  as  a 
percentage  of  maximum  performance)  and  the  transfer  of  training 
from  the  training  device  to  operational  equipment.  Transfer  of 
training  is  determined  primarily  by  comparing  task  cue  and  response 
requirements  with  device  fidelity  capabilities.  Cost  •'s  determined 
by  (a)  the  time  required  to  learn  the  task,  which,  in  turn,  is 
influenced  by  the  instructional  features  included  on  the  devices, 
and  (b)  the  hourly  estimated  costs. 

In  the  second  step,  the  individual  task  solutions  are 
combined,  and  total  costs  and  training  time  requirements  are 
calculated.  The  analysis  of  individual  tasks  makes  assumptions 
about  the  utilization  of  training  devices  that  can  only  be  verified 
after  all  tasks  have  been  assigned  to  training  devices.  If  the 
actual  utilization  is  too  different  from  the  assumptions,  the 
assumptions  may  be  changed,  and  the  analysis  repeated  at  the  user's 
option. 

Resource  Allocation  Module 


This  module  produces  a  detailed  allocation  of  training 
resources  to  the  training  devices  selected  with  the  aid  of  the 
other  tools.  The  objective  of  this  module  is  to  minimize  the  cost 
of  meeting  training  requirements.  The  actual  cost  measure 
minimized  is  the  pro  rata  per-student  total  training-system  life- 
cycle  cost  incurred  in  training  a  student  to  criterion 
performance.  The  resources  allocated  refer  to  student  training 
time  at  each  stage  of  a  course. 

The  Resource  Allocation  Module  generalizes  the  Training- 
Device  Selection  Module  in  two  ways;  (a)  It  accounts  for  the 
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discontinuities  in  the  cost  function  that  arise  from  the 
procurement  of  individual  training  devices,  (b)  it  allows  the  user 
to  specify  constraints  on  the  total  time  a  particular  training 
device  (or  actual  equipment)  may  be  used,  and  on  the  minimum 
performance  level  at  which  a  training  device  may  be  employed. 

Input  requirements.  The  input  requirements  for  this  tool 
include  all  those  for  the  Training-Device  Selection  Module.  In 
addition,  the  module  requires  the  following  constraints  specified 
by  the  user. 

1.  User  judgment  regarding  the  maximum  number  of  hours  each  type 
of  training  equipment  would  be  available  for  use.  This 
judgment  reflects  the  total  number  of  devices  available.  If 
the  user  does  not  specify  any  constraints,  then  the  number  of 
available  devices  will  not  be  limited  in  the  analysis. 

2.  User  judgment  regarding  the  minimum  performance  level  at  which 
each  device  may  be  used  for  each  task.  This  constraint 
reflects  the  fact  that  for  some  tasks,  practice  on  actual 
equipment  (or  possibly  another  device)  might  be  too  dangerous 
if  the  student  did  not  already  have  sufficient  skills. 

Module  output.  The  outputs  of  the  module  include  the  time 
each  student  should  spend  on  each  device,  the  number  of  devices 
that  must  be  acquired  to  provide  that  time,  and  the  final 
per-student  pro  rata  costs  of  training  to  criterion. 

Method.  The  method  to  solve  the  resource  allocation  problem 
is  heuristic  and  iterative.  It  operates  using  detailed  device  cost 
curves  made  up  of  several  linear  segments.  The  general  idea  is  to 
detail  the  device  cost  curves  only  around  the  solution.  The 
strategy  generates  an  initial  solution  using  relatively  undetailed 
cost  curves.  Then  the  cost  curves  are  detailed  around  the  initial 
solution  and  a  new  solution  is  found  using  the  detailed  cost 
curves.  If  the  new  solution  lies  within  those  cost  curve  domains, 
the  process  is  terminated,  and  the  solution  is  deemed  optimal.  If 
the  current  solution  does  not  lie  totally  within  current  domains, 
the  cost  curves  are  re-detailed  around  the  current  solution,  and 
a  new  cost  solution  found.  If  the  process  does  not  succeed  after 
a  predetermined  set  of  iterations,  the  process  is  terminated  and 
the  last  solution  found  is  deemed  "optimal,"  even  though  it  lies 
outside  the  domain  of  detailing  for  one  or  more  devices.  In  any 
case,  the  true  cost  functions  are  used  to  determine  the  per- 
student  cost  of  training  for  the  final  solution. 

The  procedure  for  this  module  is  carried  out  in  three  steps 
that  are  carried  out  iteratively.  In  the  first  step  the  training- 
device  cost  curves  are  represented  as  piecewise-1 inear  functions. 
Because  of  limitations  in  processing  time,  the  device  cost  curves 
are  only  partially  detailed.  In  the  second  step  the  tasks  are 
allocated  to  devices  to  minimize  the  total  per-student  training 
cost,  as  determined  by  the  current  set  of  cost  curves.  In  the 
third  step,  the  solution  is  examined  to  determine  whether  it  is 
consistent  with  the  cost  curves.  If  the  solution  is  consistent 
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with  the  cost  curves,  tne  process  is  complete.  If  an  inconsistency 
exists,  then  the  first  step  is  carried  out  again,  using  the  current 
solution  as  the  basis  for  defining  the  cost  curves. 

Summary  of  Rule  Bases 

The  numerical  calculations  performed  by  the  OSBATS  model  are 
supported  by  two  rule  bases  that  employ  reasoning  processes  that 
cannot  easily  be  expressed  numerically.  The  fidelity  rule  base 
and  the  instructional  features  rule  base  determine  the  values  of 
intermediate  parameters  used  by  the  OSBATS  model.  The  fidelity 
rule  base  generates  the  task  cue  and  response  requirements  from 
data  about  task  activities.  Similarly,  the  instructional  features 
rule  base  generates  a  matrix  that  indicates  which  instructional 
features  are  appropriate  for  each  task. 

Structure  of  the  Rule  Base 


The  rule  bases,  developed  using  the  EXSYS  Expert  System 
Development  Package  (copyright  Exsys,  Inc.),  are  composed  of  IF- 
THEN-ELSE  logical  statements  called  rules.  The  IF  section  of  a 
rule  contains  the  conditions  which  determine  whether  the  rule  will 
be  executed  and  which  section  of  the  rule  will  be  executed.  The 
rule  base  evaluates  the  conditions  in  the  IF  section  and  determines 
whether  they  are  true,  false  or  unknown.  If  the  conditions  are 
true  the  THEN  section  is  executed;  if  the  conditions  are  false  the 
ELSE  section  is  executed.  If  the  truth  of  the  conditions  is 
unknown  the  rule  base  searches  for  the  necessary  information  before 
processing  the  rule. 

The  rule  bases  operate  by  a  process  called  backward  chaining, 
meaning  that  the  rule  base  is  searched  for  needed  information 
before  the  user  is  asked  to  supply  it.  That  is,  the  rule  processor 
first  examines  rules  that  directly  produce  the  desired  conclusions. 
In  the  OSBATS  rule  bases,  these  rules  assign  tasks  to  instructional 
features  or  set  cue  and  response  requirements.  If  the  conditions 
of  the  rules  being  examined  are  known  to  be  true  or  false,  then  the 
appropriate  inferences  are  made.  If  the  truth  of  the  conditions 
is  unknown,  then  the  rule  processor  searches  for  rules  that  can  be 
used  to  derive  the  unknown  values  and  attempts  to  apply  these 
rules.  This  process  continues  until  a  rule  is  found  that  can  be 
applied,  or  until  all  relevant  rules  have  been  examined.  If 
required  information  is  still  unknown  after  all  rules  have  been 
examined,  the  user  is  asked  to  supply  the  information. 

Data  in  the  rule  bases  are  composed  of  qualifiers,  variables, 
and  choices.  A  qualifier  is  a  qualitative  variable  that  may  take 
on  one  or  more  values  from  a  specified  set.  It  is  represented  as 
a  multiple  choice  question.  Qualifiers  can  be  used  in  the  IF 
section  of  a  rule,  or  set  the  THEN  and  ELSE  sections.  Variables 
are  most  often  used  in  mathematical  and  numerical  expressions 
although  they  may  have  text  values.  Like  qualifiers,  variables  can 
be  used  in  the  IF  section,  and  set  in  the  THEN  and  ELSE  sections 
of  a  rule.  Choices  are  the  possible  solutions  and  are  assigned 
probability,  or  likelihood,  values.  For  example,  the  Instructional 
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Features  rule  base  contains  as  choices  the  twenty-one  instructional 
features.  In  this  rule  base  choices  can  be  assigned  likelihood 
values  of  zero  or  one;  in  other  words,  either  the  instructional 
feature  enhances  training  or  it  does  not. 

Instructional  Feature  Rule  Base 


The  instructional  feature  rules  are  a  direct  mapping  of  the 
task  under  consideration  to  appropriate  instructional  features. 
Data  which  the  user  must  provide  include  the  activities  involved 
in  the  task,  the  task  entry  and  standard  performance  levels,  the 
time  scaling  factor  (calculated  from  training  times) ,  the 
difficulty  of  the  task  (low,  high)  ,  the  probability  of  a  crash 
(low,  high)  and  several  other  factors  involved  in  training  a  task. 
Two  of  the  rules  used  in  the  Instructional  Features  rule  base  are 
shown  below. 

RULE  NUMBER  1: 

IF:  The  training  device  will  be  used  to  measure  performance 

THEN:  Automated  Performance  Measurement  -  Probability  =  1 


RULE  NUMBER  27: 

IF:  The  type  of  activity  involved  in  the  task  is  continuous 

movement  or  procedures  or  decision  making/rule  using 

and  Discrete  behaviors  in  this  task  are  detectable  by  computer 

THEN:  The  training  device  will  be  used  to  measure  performance 

ELSE:  The  training  device  will  not  be  used  to  measure  performance 


Because  Automated  Performance  Measurement  has  arbitrarily  been 
made  the  first  choice,  rule  number  1  is  examined  first.  The  rule 
base  then  searches  the  rules  for  information  about  whether  the 
training  device  is  used  to  measure  performance.  This  information 
is  found  in  rule  number  27.  To  process  rule  27  the  rule  base  needs 
information  about  the  type  of  activities  involved  in  the  task. 
Because  the  rule  base  contains  no  information  on  these  activities, 
the  user  must  supply  the  types  of  activities.  If  the  first 
condition  in  rule  27  is  true  then  the  second  condition  is  examined. 
The  rule  base  now  needs  information  about  the  discrete  behaviors. 
Again,  since  this  information  is  not  available  from  the  rule  base 
the  user  must  supply  it.  If  both  IF  conditions  are  determined  to 
be  true  the  rule  base  will  execute  the  THEN  section  of  rule  27  and 
then  process  rule  1.  Otherwise,  the  rule  base  will  execute  the 
ELSE  section  of  rule  27  and  skip  rule  1. 
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Fidelity  Requirement  Rule  Base 

The  fidelity  requirement  rule  base  evaluates  task  training 
and  performance  data  to  determine  the  cue  and  response  requirements 
for  each  task  across  eleven  fidelity  dimensions.  The  rules  in  this 
rule  base  are  grouped  according  to  the  fidelity  dimension  being 
evaluated.  An  explanation  of  each  group  of  rules  is  given  below, 
by  dimension  name.  The  first  seven  requirements,  visual 
resolution,  visual  content,  visual  texture,  front  and  side  field- 
of-view,  and  points  and  area  special  effects,  are  calculated  if  an 
out-of-cockpit  view  is  required.  If  no  out-of-cockpit  view  is 
required  these  requirements  are  given  the  value  zero. 

Visual  resolution.  The  visual  resolution  requirement  involves 
the  most  lengthy  calculations.  The  rule  base  calculates  the  visual 
resolution  necessary  to  perform  activities  relevant  to  the  task 
such  as  estimating  altitude,  slant  range  and  clearance,  detecting 
objects  and  landing  on  a  slope.  The  visual  resolution  requirement 
for  a  task  is  determined  by  the  activity  that  has  the  most  severe 
resolution  requirements. 

Visual  content.  The  value  of  the  visual  content  requirement 
is  determined  by  the  level  of  minimal  scene  content  that  the  user 
indicates  is  necessary  to  train  the  task  effectively. 

Visual  texture.  The  visual  texture  requirement  is  calculated 
based  on  (a)  the  size  of  the  topographical  data  base,  (b)  the 
environment  in  which  the  task  is  performed,  and  (c)  whether 
judgments  about  altitude,  slant  range  and  clearance  must  be  made 
to  perform  the  task. 

Front  f ield-of-view.  The  front  f ield-of-view  requirement  is 
calculated  based  on  information  about  peripheral  cues,  operation 
of  the  aircraft  when  severely  pitched  up  or  rolled,  whether 
identification  of  ground  locations  is  a  required  activity  and 
whether  judgments  of  clearance  must  be  made.  This  information 
determines  the  size  of  the  front  f ield-of-view.  The  size  then 
determines  the  cue  and  response  requirement. 

Side  f ield-of-view.  The  side  f ield-of-view  requirement  is 
determined  based  on  the  required  size  of  the  side  windows  and 
whether  both  side  windows  are  necessary.  The  size  of  the  side 
windows  is  dependent  on  whether  identification  of  ground  locations 
is  a  required  activity  and  whether  the  ground  locations  can  be 
viewed  through  either  the  front  or  the  side  windows. 

Special  effects  points  requirement.  The  special  effects 
requirement  is  based  on  the  content  elements  that  would  enhance 
the  effectiveness  of  training  on  the  task.  The  content  elements 
that  the  user  can  select  from  include  moving  ground  vehicles, 
airborne  vehicles,  damaged  vehicles,  weapons  blasts  and  cultural 
lights . 

Special  effects  area  requirement.  The  area  effects 
requirement  is  also  determined  by  the  content  elements  that  would 
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enhance  training  effectiveness.  The  possible  area  effects  include 
rotor  wash  and  smoke  and  dust. 

Motion  platform  requirement.  Platform  motion  is  required 
instead  of  seat  motion  when  (a)  moderate  or  high  cues  for 
performance  of  the  task  are  provided  by  one  of  the  following  two 
motions,  lateral  acceleration  or  yaw,  and  (b)  either  the  motion 
cues  are  not  correlated  with  visual  cues  or  the  motion  provides  a 
cue  for  the  initiation  of  an  emergency  procedure.  The  value  of 
the  motion  platform  cue  and  response  requirement  is  determined  by 
the  other  motions  needed  to  perform  the  task.  The  user  can  select 
from  the  following:  longitudinal  acceleration,  lateral 
acceleration,  vertical  acceleration,  yaw,  pitch  or  roll. 

Motion  seat  requirement.  Seat  motion  is  the  default  motion 
requirement  when  platform  motion  is  not  necessary.  The  seat  motion 
cue  and  response  requirement  is  calculated  based  on  (a)  which 
motions  the  user  identifies  as  providing  cues  for  performance  of 
the  task,  (b)  whether  the  type  of  activity  involved  in  the  task  is 
NOT  continuous  movement,  and  (c)  whether  prolonged  accelerations 
or  decelerations  are  required. 

Audio  effects  requirement.  Determination  of  the  audio  effects 
requirement  requires  the  user  to  identify  the  audio  signals  used 
to  perform  the  task,  indicate  whether  the  signals  are  correlated 
with  visual  cues  and  indicate  whether  they  provide  a  cue  for  the 
initiation  of  an  emergency  procedure. 

Data  base  size  requirement.  The  data  base  requirement  is 
determined  based  on  the  minimum  size  of  the  topographical  data  base 
that  the  user  indicates  is  necessary  to  perform  the  task.  The 
required  data  base  size  is  zero  if  there  is  no  visual  system, 
unless  the  task  involves  instrument  flight. 

Summary  of  Data  Requirements 

The  OSBATS  model  requires  a  considerable  amount  of  data  to 
estimate  optimal  device  designs  and  configurations.  The  specific 
data  required  and  their  format  are  derived  from  the  methods  and 
goals  of  the  five  modules.  This  section  presents  an  overview  of 
the  input  data  requirements.  The  formal  model  description  will 
specify  and  explain  in  detail  how  the  data  are  used  by  the  model. 
This  summary  briefly  defines  classes  of  data  that  are  required, 
describes  the  required  format,  and  discusses  potential  data 
sources . 

There  are  two  types  of  data  required  to  support  the 
functioning  of  the  OSBATS  model.  The  first  type  of  data,  called 
resident  or  internal  data,  cover  the  unchanging  or  slowly  changing 
information  and  relational  rules  involved  in  the  generation  of 
options,  tradeoffs,  and  configurations.  The  second  type  of  data 
required  by  the  model  is  situationally  specific  data,  the  data  used 
to  initiate  execution  of  the  models. 
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The  resident  data  include  the  rules  in  the  fidelity 
requirement  and  instructional  feature  rule  bases,  as  well  as 
descriptions  of  fidelity  dimensions  and  levels,  instructional 
features,  cost  estimates,  assumptions  used  to  determine  learning 
parameters,  and  so  forth.  The  input  data  are  used  to  initiate 
execution  of  the  models.  These  data  include  descriptions  of  the 
tasks  to  be  taught,  the  task  performance  criteria  to  be  met  by  the 
training,  the  number  of  students,  and  time  required  to  train  each 
task  on  actual  equipment. 

Although  the  model  user  is  not  considered  the  major  provider 
of  the  data  required  by  the  model,  certain  inputs  are  required  of 
the  user.  These  inputs  consist  of  the  critical  judgments  that 
express  general  priorities  in  training-system  design,  and  that 
limit  the  scope  of  the  problem  addressed  by  the  OSBATS  model.  The 
user  input  requirements  are  enumerated  below. 

1.  Weights  that  express  the  relative  importance  of  operating- 
cost  savings  as  opposed  to  safety  and  training-effectiveness 
concerns  in  determining  whether  a  task  should  be  trained  on 
a  training  device  or  on  actual  equipment. 

2.  A  value  that  reflects  the  relative  importance  of  savings  in 
investment  cost  as  opposed  to  savings  in  operating  cost. 

3.  Specification  of  whether  all  tasks  should  be  weighted  equally 
in  the  analysis,  or  whether  tasks  should  be  weighted  according 
to  the  amount  of  training  on  actual  equipment  required  to 
reach  the  training  standard. 

4.  Specification  of  whether  historical  data  on  the  likelihood 
that  instructional  features  are  used  in  existing  training 
devices  should  be  used  in  evaluating  the  benefit  of  these 
features  in  devices  that  are  being  designed. 

5.  Assumptions  that  should  be  made  about  training-device 
utilization  to  determine  the  total  hourly  cost  of  the  training 
device. 

6.  Constraints  on  the  maximum  number  of  training  devices  to  be 
procured,  and  on  the  minimum  performance  level  in  which  each 
training  device  may  be  used  for  each  task. 

7.  Limits  on  the  tasks,  training-device  options,  candidate 
instructional  features,  and  fidelity  options  that  are 
considered  in  the  analysis. 

Data  Requirements  and  Format 

The  data  requirements  for  the  OSBATS  modules  can  be  organized 
into  the  following  six  categories  with  their  respective 
subcategories . 

1.  Task  training  requirements.  This  class  of  data  includes 
information  about  the  training  requirements  associated  with 
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the  tasks  that  must  be  performed  to  prescribed  standards 
following  training.  This  class  includes  three  subclasses. 

a.  Task  learning  points.  These  data  describe  for  each  task 
student  entry  performance  level  and  performance  standard 
on  a  scale  that  ranges  from  no  knowledge  (0)  to  expert 
performance  levels  (1.0). 

b.  Task  simulation  evaluation  factors.  These  data  include  a 
rating  of  each  task  on  a  checklist  of  factors  that  are 
relevant  to  determining  the  need  for  simulation,  including 
safety  concerns,  special  performance  conditions,  and 
anticipated  training  effects. 

2.  Other  task  data.  Other  task  data  include  three  kinds  of 
information  about  tasks. 

a.  Task  training  hours  and  costs.  These  data  describe  the 
training  time  and  costs  involved  in  meeting  the  training 
requirements  for  each  task  without  a  training  device.  Data 
elements  describe  the  number  of  training  hours  required  in 
classroom,  actual  equipment  in  both  operational  and  non- 
operational  modes,  set-up  time,  and  the  cost  of  other 
required  equipment. 

b.  Task  information  processing  characteristics.  These  data 
rate  tasks  on  a  checklist  of  information-processing 
activities,  such  as  timesharing  or  continuous-control 
processes,  that  are  relevant  to  the  evaluation  of  training- 
device  instructional  features. 

c.  Task  activities.  These  data  describe  the  activities 
required  to  perform  the  task  according  to  the  variables 
required  by  the  fidelity  rules.  This  class  of  data 
encompasses  several  variables  that  are  specific  to  the  task 
and  domain. 

3.  Training-device  data.  This  class  of  data  describes 

hypothetical  or  actual  training  media  in  terms  of  cost,  cue 
and  response  capabilities,  and  instructional  features.  This 
class  includes  three  subclasses. 

a.  Training-device  costs.  These  data  include  the  following 
data  elements  for  each  training  device:  investment  cost, 
annual  fixed  operating  cost,  hourly  variable  operating 
cost,  maximum  annual  utilization,  and  training-device  life 
cycle. 

b.  Training-device  cue  and  response  capabilities.  These  data 
rate  the  technical  performance  of  each  training  device  on 
each  of  the  fidelity  dimensions  defined  in  4. a. 

c.  Training-device  instructional  features.  These  data  provide 
a  checklist  of  the  instructional  features  possessed  by  each 
training  medium. 


28 


4.  Fidelity  dimension  data.  This  class  of  data  defines  the  set 
of  options  that  are  considered  by  the  Fidelity  Optimization 
Model,  defines  the  technical  performance  scale  in  terms  of 
concrete  options,  and  contains  parameters  for  estimating 
training-device  cost  as  a  function  of  cue  and  response 
capabilities.  Fidelity  dimension  data  include  four  subclasses. 


a.  Fidelity  dimensions  and  levels.  These  data  define  each 
fidelity  dimension  and  list  the  names  of  all  levels  and 
the  associated  technical  performance  rating,  on  a  scale 
from  0  to  1.0. 

b.  Fidelity  dimension  cost  data.  This  class  of  data  includes 
the  three  parameters  of  the  function  that  is  used  to 
estimate  the  cost  of  a  particular  level  from  its  technical 
performance.  The  three  parameters  describe  the  minimum 
cost,  maximum  cost,  and  an  exponent  that  describes  the 
shape  of  the  cost  curve. 

c.  Fidelity  dimension  minimum  performance  parameter.  This 
parameter,  assessed  for  each  fidelity  dimension,  estimates 
the  transfer  of  training  that  would  occur  when  the 
capability  on  the  subject  fidelity  dimension  is  nil,  but 
capabilities  on  all  other  dimensions  are  perfect. 

d.  Fidelity  rules.  These  data  are  an  ordered  set  of 
conditional  statements  that  derive  the  task  cue  and 
response  requirements  from  a  description  of  the  activities 
required  to  perform  a  task. 

5.  Instructional-feature  data.  This  class  of  data  describes  the 
costs  and  benefits  of  the  instructional  features  and  gives 
specific  rules  for  associating  instructional  features  to  tasks. 
Included  in  this  class  of  data  are  two  subclasses. 

a.  Instructional-feature  rules.  Instructional  feature  rules 
specify  the  conditions  under  which  each  instructional 
feature  would  improve  training  efficiency.  The  conditions 
may  reference  other  elements  in  the  data  base. 

b.  Instructional-feature  cost  and  weight.  The  data  elements 
in  this  class  include  an  assessment  of  the  development  cost 
of  each  instructional  feature,  and  an  assessed  weight  that 
moderates  the  calculated  benefit  values  for  instructional 
features. 

6.  Training-system  data.  This  class  of  data  includes  a  variety 
of  miscellaneous  data  and  general  information  about  the 
training  course.  This  data  class  includes  the  following  two 
subclasses. 

a.  Course  and  system  information.  A  single  element  describing 
the  required  number  of  graduates  per  year  is  included  in 
this  category. 
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b.  Model  information.  This  class  of  data  includes  a  variety 
of  assumptions  used  by  the  model.  The  nature  of  each  data 
element  is  described  in  the  formal  model  description. 

Each  of  the  categories  described  above  consists  of  several 
data  elements.  Table  26  gives  a  brief  description  of  each  data 
element,  and  lists  the  scale  on  which  it  is  measured.  Data 
elements  will  be  referenced  by  the  category  number  listed  in  the 
table.  For  example,  the  set  of  performance  standards  will  be 
referred  to  as  "lAl.”  Similarly,  a  class  of  data  elements  will 
receive  the  appropriate  number  of  the  general  data  category  for 
that  class.  For  example,  task  training  requirements  will  be 
referred  to  by  the  category  "1.”  Also  included  in  Table  26  are 
potential  sources  of  the  data;  the  nature  of  these  sources  is 
described  in  the  following  paragraphs. 

Data  Sources 


The  data  required  to  operate  the  OSBATS  model  will  come  from 
several  sources,  including: 

1.  subject-matter  experts  (SMEs) , 

2.  training-system  experts  (TSEs) , 

3.  training  researchers  (TRs) , 

4.  model  developers  (MDs) ,  and 

5.  model  users  (MUs) . 

As  the  model  evolves,  we  expect  the  nature  of  the  data  required 
from  experts  to  change,  with  subject-matter  and  training-system 
experts  providing  simpler  judgments  that  are  more  factual  and  less 
subjective  These  judgments  would  be  transformed  to  produce  the 
data  required  by  the  model.  In  the  near  term,  however,  experts 
will  be  required  to  provide  a  variety  of  judgmental  data  to  meet 
the  model  requirements. 

The  likely  source  of  each  data  element  is  shown  in  Table  1. 
General  descriptions  of  these  data  sources  are  given  below. 

1.  Subject  matter  experts  include  instructors,  training 
developers,  and  expert  job  performers.  These  experts  are 
characterized  by  their  knowledge  of  the  tasks  being  trained. 
They  are  the  primary  source  of  task  training  requirement  and 
other  task  data. 

2.  Training-system  experts  are  characterized  by  their  knowledge 
of  the  capabilities  and  costs  of  training  devices.  They  are 
the  primary  source  of  training-device  data,  fidelity  dimension 
data,  and  instructional-feature  cost  data. 

3.  Training  researchers  provide  the  link  between  the  model  and 
the  body  of  relevant  behavioral  research.  They  will  be  the 
major  source  of  instructional-feature  data.  In  addition, 
training  research  will  play  an  important  part  in  the  form  of 
the  functions  used  to  predict  training  cost  and  effectiveness. 
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Table  1 


Description  of  OSBATS  Model  Data  Requirements 


DB 

Format/ 

Cat. 

Description 

Scale  Source 

1 

Task  Training  Requirements 

lA 

Task  Learning  Points 

lAl 

Entry  Performance  Level 

[0,1.0] 

SME 

1A2 

Performance  Standard 

[0,1.0] 

SME 

IB 

Task  Simulation  Evaluation  Factors 

IBl 

Absolute  Requirement 

{0,1} 

SME, MU 

1B2 

Special  Weather 

{0,1} 

SME, MU 

1B3 

Special  Situation 

{0,1} 

SME, MU 

1B4 

Special  Equipment 

{0,1} 

SME, MU 

IBS 

Training  Effectiveness  Enhancements 

{0,1} 

SME, MU 

2 

Other  Task  Data 

2A 

Task  Training  Hours  and  Costs 

2A1 

Classroom  Hours 

hours/grad 

SME 

2A2 

Actual  Equip  Non-Operational  Hours 

hours/grad 

SME 

2A3 

Actual  Equipment  Operational  Hours 

hours/grad 

SME 

2A4 

Actual  Equipment  Setup  Hours 

hours/grad 

SME 

2A5 

Other  Training  Costs 

$/grad 

SME 

2B 

Task  Info  Processing  Characteristics 

2B1 

Continuous  Movement 

{0,1} 

SME,TR 

2B2 

Procedural 

{0,1} 

SME,TR 

2B3 

Perception 

{0,1} 

SME , TR 

2B4 

Decision  Making/Rule  Using 

{0,1} 

SME , TR 

2B5 

Verbal  Information 

{0,1} 

SME , TR 

2B6 

Voice  Communication 

{0,1} 

SME , TR 

2B7 

More  Than  Seven  Steps 

{0,1} 

SME , TR 

2B8 

Time  Sharing 

{0,1} 

SME , TR 

2B9 

Computer-Detectable  Responses 

{0,1} 

SME , TR 

2B10 

Meaningful  Performance  Tolerances 

{0,1} 

SME , TR 

2B11 

Intrinsic  Feedback 

{0,1} 

SME , TR 

2B12 

Cue  Salience 

{0,1} 

SME , TR 

2B13 

Crash  Probability 

{0,1} 

SME , TR 

2B14 

Situational  Awareness 

{0,1} 

SME , TR 

2C 

Task  Activities 

various 

SME 

3 

Training  Device  Data 

3A 

Training  Device  Costs 

3A1 

Investment  Cost 

$/device 

TSE 

3A2 

Annual  Fixed  Operating  Cost 

$/device/yr  TSE 

3A3 

Hourly  Variable  Operating  Cost 

$/grad/hr 

TSE 

3A4 

Lifecycle 

years 

TSE, MU 

3A5 

Maximum  Annual  Utilization 

hrs/yr 

TSE, SME 
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Table  1  (Continued) 


Description  of  OSBATS  Model  Data  Requirements 


DB 

Format/ 

Cat. 

Description 

Scale 

Source 

3B 

Training  Device  Cue  & 

Response  Capabilities 

3B1 

Visual  Resolution 

[0,1.0] 

TSE 

3B2 

Visual  Content 

[0,1.0] 

TSE 

3B3 

Visual  Texture 

[0,1.0] 

TSE 

3B4 

Front  Visual  Field  of  View  (FOV) 

[0,1.0] 

TSE 

3B5 

Side  Visual  FOV 

[0,1.0] 

TSE 

3B6 

Point  Special  Effects 

[0,1.0] 

TSE 

3B7 

Area  Special  Effects 

[0,1.0] 

TSE 

3B8 

Platform  Motion 

[0,1.0] 

TSE 

3B9 

Seat  Motion 

[0,1.0] 

TSE 

3B10 

Sound  Special  Effects 

[0,1.0] 

TSE 

3B11 

Map  Area 

[0,1.0] 

TSE 

3C 

Training-Device  Instructional  Features 

3C1 

Automated  Performance  Measurement 

{0,1} 

TSE 

3C2 

Performance  Indicators 

{0,1} 

TSE 

3C3 

Procedure  Monitoring 

{0,1} 

TSE 

3C4 

Automated  Performance  Alerts 

{0,1} 

TSE 

3C5 

Augmented  Feedback 

{0,1} 

TSE 

3C6 

Augmented  Cues 

{0,1} 

TSE 

3C7 

Record/Playback 

{0,1} 

TSE 

3C8 

Total  System  Freeze 

{0,1} 

TSE 

3C9 

Remote  Graphics  Replay 

{0,1} 

TSE 

3C10 

Initial  Conditions 

{0,1} 

TSE 

3C11 

Scenario  Control 

{0,1} 

TSE 

3C12 

Crash  Override 

{0,1} 

TSE 

3C13 

Reset/Reposition 

{0,1} 

TSE 

3C14 

Parameter  Freeze 

{0,1} 

TSE 

3C15 

Flight  System  Freeze 

{0,1} 

TSE 

3C16 

Positional  Freeze 

{0,1} 

TSE 

3C17 

Real-Time  Variables  Control 

{0,1} 

TSE 

3C18 

Automated  Simulator  Demonstration 

{0,1} 

TSE 

3C19 

Adjunct  CAI 

{0,1} 

TSE 

3C20 

Automated  Adaptive  Training 

{0,1} 

TSE 

3C21 

Automated  Cueing  and  Coaching 

{0,1} 

TSE 

4 

Fidelity  Dimension  Data 

4A 

Fidelity  Dimensions  and  Levels 

4A1 

Technical  Performance  Levels 

[0,1.0] 

MD,TSE,SME 

4B 

Fidelity  Dimension  Cost  Data 

4B1 

Minimum  Cost 

investment 

$  TSE,MD 

4B2 

Maximum  Cost 

investment 

$  TSE,MD 

4B3 

Exponent 

>0 

TSE , MD 

4C 

Minimum  Performance  Parameter 

[0,1.0] 

TR,TSE,SME 

4D 

Fidelity  Rules 

not  numeric  TR,MD,SME 
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Table  1  (Continued) 

Description  of  OSBATS  Model  Data  Requirements 


DB 

Cat. 

Description 

Format/ 

Scale  Source 

5 

5A 

Instructional  Feature  Data 
Instructional  Feature  Rules 

not  numeric 

TR,MD 

5B 

Instructional  Feature  Cost  and  Weight 

5B1 

Instructional  Feature  Cost 

investment  $ 

TSE 

5B2 

Instructional  Feature  Benefit  Weight 

[0,1.0] 

TR,SME 

6 

6A 

6A1 

Training  System  Data 

Course  and  System  Information 

Annual  Student  Throughput 

grads/yr 

SME,MU 

6B 

6B1 

Model  Information 

Standard  Adjustment  Rate 

% 

TR,MD 

6B2 

Learning  Curve  Exponent 

>0 

TR,MD 

6B3 

Maximum  Instructional  Feature  Effect 

% 

TR,MD 

6B4 

Maximum  Number  of  Instructional 
Features 

{1,2,3,...} 

TR,MD 

6B5 

Assumed  Setup  Savings  Percentage 

% 

TR,MD 

6B6 

Cost  Savings  Weight 

[0,1.0] 

MU 

6B7 

Recommend ucinn  Boundaries 

[0,1.0] 

MD 

6B8 

Annual  Fixed  Cost  Factor 

$inv/$ fixed 

TSE,MD 

6B9 

Hourly  Variable  Cost  Factor 

$inv/$var 

TSE,MD 

6B10 

New  Device  Utilization 

hours/year 

TSE,MD 

6B11 

New  Device  Lifecycle 

years 

TSE,MD 
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4. 


Model  developers  are  required  to  produce  data  in  some  of  the 
areas  in  which  consideration  of  the  model  form  is  required. 
For  example,  the  fidelity  optimization  module  assumes  that 
cost  and  benefit  of  fidelity  dimensions  are  mutually 
independent.  The  model  developer  will  be  able  to  structure 
the  fidelity  dimensions  to  reduce  or  eliminate  the  effect  of 
any  interactions.  Consequently,  we  expect  that  tne  moael 
developer  will  work  with  the  training-system  expert  to  define 
the  fidelity  dimensions  and  levels. 

5.  The  model  user  may  be  one  of  the  other  four  kinds  of  experts, 
or  may  be  a  project  manager  who  must  aggregate  the  specific 
expertise  of  members  of  his  staff.  Although  the  user  will 
have  access  to  the  data  base,  we  expect  the  user  to  have  three 
major  impacts:  (a)  to  make  the  value  judgments  that  affect  the 
critical  weights  used  at  various  points  in  the  analysis,  (b) 
to  set  the  scope  of  the  analysis,  and  (c)  to  adjust  the 
results  of  the  analysis  to  account  for  factors  that  are  not 
included  in  the  model. 

Summary  of  Input  Requirements  for  Individual  Modules 

Table  2  summarizes  the  input  requirements  for  the  five  OSBATS 
modules.  For  each  data  element,  it  indicates  the  modules  for  which 
the  information  is  required. 

Summary  of  Model  Assumptions 

The  OSBATS  model  makes  several  assumptions  regarding  the 
training  process.  Some  of  these  assumptions  are  central  to  the 
model's  description  of  the  training  process.  Other  assumptions 
simplify  the  problem  so  that  it  can  be  addressed  efficiently  with 
minimal  input  data.  Still  other  assumptions  represent  conjectures 
about  aspects  of  the  training  process  for  which  there  is  little 
empirical  data.  Details  of  these  assumptions  are  given  in  the 
formal  system  description.  This  section  describes  some  of  the  more 
important  model  assumptions. 

The  model  assumptions  are  organized  into  the  following  four 
classes. 

1.  Assumptions  that  specify  the  functional  relationship  between 
variables,  such  the  relationship  between  performance  and  amount 
of  training. 

2.  Assumptions  that  specify  that  certain  variables  are  independent 
of  other  variables. 

3.  Assumptions  that  deal  with  the  model's  treatment  of  cost. 

4.  Specific  assumptions  about  parameter  values  and  measurement 
scales . 

These  classes  of  assumption  are  described  in  the  following 
subsections. 
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Table  2 


Module  Use  of  Data  Elements 


Optimization  Module 


Resource  Allocation _ 

Training-Device  Selection _  \ 

Fidelity  Optimization  _  \  \  \ 

Instr.  Feature-Selection _  \  \  \  \ 

Simulation  Configuration _  \  \  \  \  \ 
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Table  2  (Continued) 

Module  Use  of  Data  Elements 


Optimization  Module 

\ 

Resource  Allocation _  \ 

Training-Device  Selection _  \  \ 

Fidelity  Optimization  _  \  \  \ 

Instr.  Feature-Selection _  \  \  \  \ 

Simulation  Configuration _  \  \  \  \  \ 

DB  \  \  \  \  \  \ 

Cat.  Description _ !SC  IlFSiFO  ItdsIra  ! 


1  I  I  !  T 


3 

Training  Device  Data 

X 

1 

1 

X 

1 

1 

X 

1 

1 

X 

1 

1 

X 

3A 

Training  Device  Costs 

X 

1 

1 

X 

1 

1 

X 

1 

1 

X 

1 

1 

X 

3A1 

Investment  Cost 

X 

1 

1 

X 

1 

1 

X 

1 

1 

X 

1 

1 

X 

3A2 

Annual  Fixed  Operating  Cost 

X 

1 

1 

X 

1 

1 

X 

1 

1 

X 

1 

1 

X 

3A3 

Hourly  Variable  Operating  Cost 

X 

1 

1 

X 

1 

1 

X 

1 

1 

X 

1 

1 

X 

3A4 

Lifecycle 

X 

1 

1 

X 

1 

1 

X 

1 

1 

X 

1 

1 

X 

3A5 

Maximum  Annual  Utilization 

1  ^ 

1 

1 

X 

1 

1 

X 

1 

X 

1 

j 

X 

3B 

Training  Device  Cue  & 

Response  Capabilities 

1  X 

1 

1 

1 

1 

1 

1 

X 

1 

1 

X 

3B1 

Visual  Resolution 

X 

1 

1 

1 

1 

1 

1 

X 

1 

1 

X 

3B2 

Visual  Content 

X 

1 

1 

1 

1 

1 

1 

X 

1 

1 

X 

3B3 

Visual  Texture 

X 

1 

1 

1 

1 

1 

1 

X 

1 

1 

X 

3B4 

Front  Visual  Field  of  View  (FOV) 

X 

1 

1 

1 

1 

1 

1 

X 

1 

1 

X 

3B5 

Side  Visual  FOV 

X 

1 

1 

1 

1 

1 

1 

X 

1 

1 

X 

3B6 

Point  Special  Effects 

X 

1 

1 

1 

1 

1 

1 

X 

1 

1 

X 

3B7 

Area  Special  Effects 

X 

1 

1 

1 

1 

1 

1 

X 

1 

1 

X 

3B8 

Platform  Motion 

X 

1 

1 

1 

1 

1 

1 

X 

1 

1 

X 

3B9 

Seat  Motion 

X 

1 

1 

1 

1 

1 

1 

X 

1 

1 

X 

3B10 

Sound  Special  Effects 

X 

1 

1 

1 

1 

1 

1 

X 

1 

1 

X 

3B11 

Map  Area 

X 

1 

1 

1 

1 

1 

1 

X 

1 

1 

X 

3C 

Training-Device  Instructional  Featuresx 

1 

1 

1 

1 

1 

1 

X 

1 

1 

X 

3C1 

Automated  Performance  Measurement 

1  X 

1 

1 

1 

1 

1 

1 

X 

1 

1 

X 

3C2 

Performance  Indicators 

X 

1 

1 

1 

1 

1 

1 

X 

1 

1 

X 

3C3 

Procedure  Monitoring 

X 

1 

1 

1 

1 

1 

1 

X 

1 

1 

X 

3C4 

Automated  Performance  Alerts 

X 

1 

1 

1 

1 

I 

1 

X 

1 

1 

X 

3C5 

Augmented  Feedback 

X 

1 

1 

1 

1 

1 

1 

X 

1 

1 

X 

3C6 

Augmented  Cues 

X 

1 

1 

1 

1 

1 

1 

X 

1 

1 

X 

3C7 

Record/Playback 

X 

1 

1 

1 

1 

1 

1 

X 

1 

1 

X 

3C8 

Total  System  Freeze 

X 

1 

1 

1 

1 

1 

1 

X 

1 

1 

X 

3C9 

Remote  Graphics  Replay 

X 

1 

1 

1 

1 

1 

1 

X 

1 

1 

X 

3C10 

Initial  Conditions 

X 

1 

1 

1 

1 

1 

1 

X 

1 

1 

X 

3C11 

Scenario  Control 

X 

1 

1 

1 

1 

1 

1 

X 

1 

1 

X 

3C12 

Crash  Override 

X 

1 

1 

1 

1 

1 

1 

X 

1 

1 

X 

3C13 

Reset/Reposition 

X 

1 

1 

1 

1 

1 

1 

X 

1 

1 

X 

3C14 

Parameter  Freeze 

X 

1 

1 

1 

1 

1 

1 

X 

1 

1 

X 

3C15 

Flight  System  Freeze 

X 

1 

1 

1 

1 

1 

1 

X 

1 

1 

X 

3C16 

Positional  Freeze 

X 

1 

1 

1 

1 

1 

1 

X 

1 

1 

X 

3C17 

Real-Time  Variables  Control 

X 

1 

1 

1 

1 

1 

1 

X 

1 

1 

X 

3C18 

Automated  Simulator  Demonstration 

X 

1 

1 

1 

1 

1 

1 

X 

1 

1 

X 
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Table  2  (Continued) 

Module  Use  of  Data  Elements 


Optimization  Module 


Resource  Allocation 


\ 


DB 


\ 

Training-Device  Selection _  \  \ 

_  \  \  \ 

.  \  \  \  \ 

\  \  \  \  \ 


Fidelity  Optimization 

Instr.  Feature-Selection _ 

Simulation  Configuration _ 


\  \  \  \  \  \ 


Cat. 

Descriotion  ! 

SC 

!  IFS 

FO 

!tds 

RA  1 

3C19 

Adjunct  CAI 

X 

1 

1 

1 

1 

1  - 

1 

X  1 
*  1 

3C20 

Automated  Adaptive  Training 

X 

1 

1 

X  ' 
^  1 

3C21 

Automated  Cueing  and  Coaching  [ 

X 

1 

1 

1  X 

X  ' 
*  1 

4 

Fidelity  Dimension  Data  [ 

X 

i 

1 

X 

1 

X  ' 

4A 

Fidelity  Dimensions  and  Levels 

X 

1 

1 

X 

1 

1 

1 

1 

4A1 

Technical  Performance  Levels 

X 

1 

1 

X 

1 

1 

1 

1 

4B 

Fidelity  Di.iicnsion  Cost  Data 

X 

1 

1 

X 

1 

1 

1 

1 

4B1 

Minimum  Cost 

X 

1 

1 

X 

1 

1 

1 

1 

4B2 

Maximum  Cost 

X 

1 

1 

X 

1 

1 

1 

1 

4B3 

Exponent 

X 

1 

1 

X 

1 

1 

1 

1 

4C 

Minimum  Performance  Parameter 

X 

I 

1 

4D 

Fidelity  Rules  } 

X 

1 

1 

X 

!  X 

X 

X  , 

5 

Instructional  Feature  Data  j 

X 

1  ^ 

1  ^ 

X  ' 
^  1 

5A 

Instructional  Feature  Rules  } 

X 

V  1 
^  1 

5B 

Instructional  Feature  Cost  and  Weight 

1 

1 

1 

1 

5B1 

Instructional  Feature  Cost  { 

1 

1 

1 

5B2 

Instructional  Feature  Benefit  Weight} 

!  X 

1 

1 

1 

1 

6 

Training  System  Data  ! 

X 

1 

1 

1  ^ 

X  ' 
*  1 

6A 

Course  and  System  Information 

X 

1 

I 

X 

^  1 

6A1 

Annual  Student  Throughput 

X 

1 

1 

X 

X  • 
*  1 

6B 

Model  Information 

X 

1 

1 

^  1 

6B1 

Standard  Adjustment  Rate 

X 

1 

1 

X  ' 
^  1 

6B2 

Learning  Curve  Exponent 

X 

1 

1 

X 

X 

^  I 

6B3 

Maximum  Instructional  Feature  Effect} 

X 

1 

1 

1  X 

X  1 

6B4 

Maximum  Number  of  Instructional 

Features  j 

1 

1 

1 

1 

1 

1 

6B5 

Assumed  Setup  Savings  Percentage 

X 

1 

1 

X 

^  1 

6B6 

Cost  Savings  Weight 

X 

1 

1 

1 

1 

1 

1 

6B7 

Recommendation  Boundaries 

X 

1 

1 

1 

1 

1 

1 

6B8 

Annual  Fixed  Cost  Factor 

1 

1 

X 

1 

1 

1 

1 

6B9 

Hourly  Variable  Cost  Factor 

1 

1 

X 

1 

1 

1 

1 

6B10 

New  Device  Utilization 

1 

1 

X 

1 

1 

1 

1 

6B11 

New  Device  Lifecycle  } 

1 

1 

X 

1 

1 

1 

1 
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Functional  Relationships 


The  OSBATS  model  includes  several  functions  that  relate  cost 
and  benefit  measures  to  training  time,  task  characteristics,  and 
training-device  features.  This  section  gives  a  verbal  description 
of  tiiese  assumed  relationships.  A  mathematical  statement  of  the 


assumptions 
Appendix  B. 

is 

given 

in  the 

formal 

model  description 

and  in 

Central 

to 

the 

Training- 

•Device 

Selection  Module 

and  the 

Resource  Allocation  Module  is  a  relationship  between  performance 
on  actual  equipment  and  training  time  (on  a  training-device  or  on 
actual  equipment) .  This  relationship  is  also  used  in  the 
Simulation  Configuration  Module.  The  OSBATS  model  assumes  that 
this  relationship  is  characterized  by  a  power  function.  Newell 
and  Rosenbloom  (1981)  have  reviewed  a  large  body  of  literature 
indicating  that  a  power  function  provides  a  good  fit  for  data  for 
which  the  time  to  complete  a  task  is  related  to  the  number  of 
practice  trials.  A  mathematical  analysis  of  the  power  function 
indicates  that  it  is  equally  appropriate  when  training  is  measured 
by  time  instead  of  trials.  We  have  extended  the  power  function  to 
apply  to  a  more  general  measure  of  performance.  The  resulting 
function  incorporates  learning  on  the  device  in  which  training  is 
conducted,  and  subsequent  transfer  of  the  training  to  actual 
equipment. 

A  critical  parameter  of  the  performance  function  is  its 
asymptote,  which  represents  the  maximum  extent  to  which  training 
on  a  training  device  will  transfer  to  actual  equipment.  Transfer 
of  training  depends  on  the  training  device  being  used  to  provide 
the  cue  and  response  requirements  of  the  tasks  being  trained. 
Thus,  it  a  function  of  training  device  fidelity.  The  OSBATS  model 
treats  this  relationship  as  a  product  of  training-device  capability 
measures  over  the  fidelity  dimensions.  The  factors  of  the  product 
function  are  assessed  so  that  different  fidelity  dimensions  receive 
different  importance  weights  in  the  relationship,  and  so  that 
fidelity  dimensions  for  which  a  task  requirement  is  high  are  more 
important  in  the  assessment  for  that  task  than  dimensions  in  which 
the  requirement  is  low.  The  asymptote  function  includes 
normalization  constants  that  ensure  that  the  results  of  the 
function  are  in  the  correct  range,  and  are  consistent  with  the 
assessed  cue  and  response  requirements. 

The  Fidelity  Optimization  Module  and  Simulation  Configuration 
Module  rely  on  a  function  that  estimates  training-device 
development  cost  from  training-device  fidelity  and  instructional 
features.  A  critical  component  of  this  function  estimates  the  cost 
of  reaching  a  particular  technical  performance  level  on  a  given 
fidelity  dimension.  The  OSBATS  model  characterizes  the 
development-cost  component  for  a  single  fidelity  dimension  as  a 
logarithmic  function  of  technical  performance.  This  function  has 
been  formulated  to  have  the  reasonable  property  that  increments  in 
technical  performance  at  a  high  level  of  fidelity  are  more 
expensive  than  increases  at  a  low  level  of  fidelity.  Each  fidelity 


38 


dimension  is  described  by  a  single  number  characterizing  the 
specific  shape  of  the  cost  function. 

The  Training-Device  Selection  Module  and  Resource  Allocation 
Module  muot  have  a  measure  of  the  hourly  life  cycle  cost  of 
training  devices.  The  Training-Device  Selection  Module  considers 
a  single  hourly  rate  for  each  training  device,  combining 
investment,  fixed  operating,  and  variable  operating  cost  components 
and  dividing  by  the  estimated  total  use  of  the  training  device  over 
its  life  cycle.  A  single  cost  rate  leads  to  a  cumulative  cost  that 
is  a  linear  function  of  the  number  of  hours  that  the  training 
device  is  used.  The  Resource  Allocation  Modules  uses  a  piecewise 
linear  cumulative  cost  function  with  discontinuities  at  the  levels 
of  use  that  require  the  procurement  of  additional  training  devices. 


Independence  Assumptions 

Independence  assumptions  simplify  the  OSBATS  model  by  ignoring 
some  of  the  many  interactions  that  are  possible  between  training 
system  variables.  To  the  extent  that  the  assumptions  are 
reasonable,  independence  can  simplify  the  analysis  process  and  add 
to  the  clarity  of  the  results.  However,  if  the  assumptions  are  not 
correct,  then  the  method  will  not  identify  the  best  solutions.  We 
have  assumed  independence  in  several  areas,  as  enumerated  below: 

1.  The  cost  and  benefit  of  any  specific  instructional  feature  does 
not  depend  on  which  other  instructional  features  are  present 
in  a  training  device. 

2.  The  cost  and  benefit  of  a  particular  level  of  a  fidelity 
dimension  does  not  depend  on  the  levels  of  other  dimensions  in 
a  training  device. 

3.  The  fidelity  of  a  training  device  is  assumed  to  affect  the 
asymptote  of  the  performance  function  (transfer  of  training) 
without  affecting  the  rate  at  which  performance  approaches  that 
asymptote . 

4.  The  instructional  features  of  a  training  device  are  assumed  to 
affect  the  rate  at  which  the  performance  approaches  the 
asymptote  of  the  performance  function,  but  does  not  affect  the 
asymptote  itself. 

5.  The  effectiveness  of  a  training  device  for  a  particular  task 
depends  only  on  the  performance  level  attained  before  the 
device  was  used,  and  does  not  depend  on  which  device  or  devices 
were  used  to  attain  that  performance  level. 

6.  Effects  of  the  sequence  in  which  tasks  are  trained  are  not 
considered  in  the  OSBATS  model.  The  assessed  tine  required 
for  training  assumes  that  tasks  are  trained  in  a  reasonable 
order. 
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Cost  Component  Assumptions 


Assumptions  of  specific  cost  estimating  functions  were 
discussed  previously.  The  major  concern  of  this  sec;:ion  is  how 
the  different  modules  treat  the  one-time  and  recurring  components 
of  life  cycle  cost.  In  this  respect,  there  are  differences  between 
the  modules  that  stem  from  differences  in  the  aspect  of  the  overall 
problem  that  they  are  addressing.  The  Training-Device  Selection 
and  Resource  Allocation  Modules  are  directly  concerned  with 
minimizing  the  life  cycle  cost  of  training.  Consequently,  these 
modules  use  life  cycle  cost  as  their  measure  of  cost.  The 
Instructional  Feature  Selection  and  Fidelity  Optimization  Modules 
are  concerned  with  the  selection  of  training-device  components. 
The  development  cost  of  each  component  may  be  specified  in  advance. 
However,  the  operating  cost  depends  on  the  number  of  tasks  for 
which  the  component  would  be  required.  Thus  for  these  modules, 
operating  cost  is  confounded  with  benefit.  Consequently,  the 
Instructional  Feature  Selection  and  Fidelity  Optimization  Modules 
consider  development  cost  as  the  major  cost  measure.  The 
Simulation  Configuration  Module  considers  both  development  cost  and 
operating  cost  in  its  analysis.  Development  cost  is  the  major  cost 
element,  while  operating  cost  savings  is  explicitly  included  as  one 
of  the  benefits  of  simulated  training. 

Specific  Assumptions 

The  OSBATS  model  requires  estimates  for  several  parameters 
for  its  operation.  Also  included  among  these  assumptions  are 
scaling  assumptions  on  the  measurement  of  benefit.  The  specific 
assumptions  are  the  following: 

1.  In  the  Simulation  Configuration,  Instructional  Feature 
Selection,  and  Fidelity  Optimization  Modules,  benefit  is 
measured  on  a  ratio  scale.  This  assumption  allows  the  model 
to  set  the  scale  of  benefit  so  that  it  varies  from  0.0  to  1.0 
among  the  options  being  considered  in  the  particular  analysis. 


2.  The  model  includes  parameters  stating  the  maximum  number  of 
instructional  features  that  would  be  useful  in  training  a 
single  task  and  the  maximum  percentage  improvement  in  training 
time  that  could  be  obtained  from  the  inclusion  of  instructional 
features.  These  values  are  currently  assumed  to  be  three 
features  and  10%,  respectively. 

3.  The  model  includes  a  parameter  expressing  the  percentage  of 
setup  time  that  can  be  saved  by  training  in  a  simulated 
environment.  That  number  is  currently  assumed  to  be  60%. 

Whereas  many  of  the  functional  relationships  described  previously 
represent  assumptions  at  the  heart  of  the  OSBATS  model,  these 
specific  assumptions  represent  specific  parameter  values  that  could 
be  changed  with  minimal  effects  on  model  operation. 
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OSBATS  MODEL  DESCRIPTION 


The  detailed  model  description  is  presented  using  IDEFO 
(Integrated  Computer  Aided  Manufacturing  Definition) .  The  IDEFO 
system  was  developed  by  the  Integrated  Computer  Aided 
Manufacturing  Office  (ICAM)  of  the  U.S.  Air  Force  to  be  used  as  a 
tool  for  describing  the  functions  and  data  of  a  complex  system 
(SofTech,  Inc.,  1981;  Ross  &  Schoroan,  1977).  In  this  report, 
IDEFO  is  used  to  describe  the  components  and  functions  of  the 
implemented  OSBATS  model.  The  first  part  of  this  section 
provides  a  conceptual  description  of  the  IDEFO  system.  The  next 
portion  provides  tables  that  summarize  information  about  the 
model  nodes,  input  and  control  data,  and  process  outputs.  Also 
included  is  a  list  of  the  symbols  used  to  represent  the  variables 
of  the  model.  Finally,  the  IDEFO  description  of  the  model 
components  are  presented  in  an  organized  node  structure. 

IDEFO  Methodology  Description 

A  system  consists  of  any  combination  of  machinery  (hardware) , 
data,  and  people,  working  together  to  perform  a  useful  function. 
IDEFO  is  a  technique  that  enables  people  to  understand  complex 
systems  and  to  communicate  their  understanding  to  others.  IDEFO 
describes  the  functions  performed  by  the  system  by  successively 
decomposing  the  system  into  its  basic  components,  describing  how 
each  component  processes  information,  and  specifying  how 
different  components  interact.  An  IDEFO  model  is  expressed  as  a 
series  of  related  diagrams;  each  diagram  describes  a  particular 
system  component  or  function.  An  IDEFO  diagram  is  composed  of 
boxes  and  arrows.  The  boxes  represent  component  functions  or 
activities,  while  the  arrows  represent  data  that  affect  the 
activities  or  are  produced  by  them. 

IDEFO  Model  Organization 

The  diagrams  in  an  IDEFO  model  describe  the  system  in  a 
modular,  top-down  fashion,  showing  the  breakdown  of  the  system 
into  its  component  parts.  The  application  of  IDEFO  starts  with 
the  most  general  or  abstract  description  of  the  system  to  be 
produced.  This  description  is  represented  in  a  diagram  as  a 
single  box;  that  box  is  subsequently  broken  down  into  a  number  of 
more  detailed  boxes,  each  of  which  represents  a  component  part. 
The  component  parts  are  then  detailed,  each  on  another  diagram. 
Each  part  shown  on  a  detail  diagram  is  again  broken  down,  and  so 
forth,  until  the  system  is  described  to  the  desired  level  of 
detail.  Lower-level  diagrams,  then,  are  detailed  breakdowns  of 
higher-level  diagrams.  At  each  stage  of  breaking  down  the 
system,  the  higher-level  diagram  is  said  to  be  the  "parent"  or 
overview  of  the  lower-level  "detail"  diagrams.  The  relationship 
between  diagrams  at  different  levels  is  shown  in  Figure  1. 

Diagram  display  format.  In  this  document,  each  diagram  in  an 
IDEFO  model  is  displayed  in  a  two-page  format.  The  subject 
diagram  is  shown  on  the  top  of  the  right-hand  page.  The  parent 
of  the  subject  diagram  is  shown  on  the  top  of  the  left-hand  page 
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Figure  1.  Example 


with  the  location  of  the  subject  node  indicated.  On  the  bottom 
half  of  both  pages  is  text  describing  the  operations  performed  by 
each  activity  represented  in  the  diagram.  Each  pair  of  pages 
receives  a  page  number  that  is  displayed  as  part  of  the  subject 
diagram. 

Diagram  node  numbers.  In  an  IDEFO  diagram,  the  component 
parts  are  shown  as  numbered  boxes.  A  diagram  should  have  no  more 
than  six  boxes.  Each  box  at  one  level  is  detailed  in  one  diagram 
at  the  next  lower  level  until  a  sufficient  level  of  detail  is 
reached.  The  place  of  each  diagram  in  a  model  is  indicated  by  a 
"node  number"  derived  from  the  numbering  of  boxes.  For  example, 
A21  is  the  diagram  that  details  box  1  on  the  A2  diagram. 
Similarly,  A2  details  box  2  on  the  AO  diagram,  which  is  the  top 
diagram  of  the  model.  The  parent  of  the  AO  diagram  represents 
the  system  as  a  single  box  and  is  denoted  "A-0."  The  hierarchy 
may  be  shown  in  an  index  of  diagram  names  and  their  node  numbers 
called  a  "node  list."  The  node  list  serves  as  a  table  of 
contents  for  a  model.  In  an  IDEFO  model,  diagrams  are  displayed 
according  to  the  order  of  their  node  numbers. 

The  example  shown  in  Figure  6  provides  an  illustration  of  the 
hierarchical  decomposition  of  functions.  The  diagrams  in  Figure 
6  indicate  that  the  overall  function,  develop  system  (AO) ,  is 
broken  down  into  three  sub-functions,  A1  through  A3.  Design 
system  (A2)  is  further  broken  down  into  three,  more  detailed 
sub-functions  (A21  through  A23) . 

Description  of  Individual  IDEFO  Diagrams 

In  IDEFO,  boxes  represent  components  in  the  breakdown,  and 
arrows  represent  relationships  between  these  components. 
Descriptive  labels  are  written  inside  each  box  and  along  each 
arrow  to  describe  their  meaning.  The  notation  is  kept  simple  to 
permit  easy  reading  with  little  special  training. 

Figure  7  shows  a  sample  IDEFO  diagram.  Notice  that  the  boxes 
represent  the  breakdown  of  activities  or  functions  performed  by 
the  system  and  are  named  by  verbs.  Arrows,  which  represent 
objects  or  information,  are  labeled  with  nouns. 

Box-and-arrow  syntax.  The  sample  IDEFO  diagram  in  Figure  7 
shows  that  the  descriptive  names  and  labels  convey  the  box  and 
arrow  contents  to  the  reader.  In  addition  to  its  label,  the  side 
at  which  an  arrow  enters  or  leaves  a  box  shows  its  role  as  an 
input,  control,  output,  or  mechanism  for  the  box  (see  Figure  8) . 
Arrows  that  enter  from  the  left  of  an  activity  box  represent 
inputs  to  the  process  represented  by  the  box.  Inputs  represent 
the  raw  materials  or  data  used  by  the  activity  to  produce 
outputs.  The  outputs  are  represented  by  arrows  that  originate 
from  the  right  side  of  the  box.  Arrows  entering  a  box  from  the 
top  represent  controls  on  the  activity.  Controls  are  data  that 
provide  catalysts  or  constraints  for  the  represented  activity, 
but  are  not  changed  by  the  process.  Finally,  arrows  that  enter  a 
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Figure  2.  IDEFO  node  numbering  convention. 
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Figure  3.  Sample  IDEFO  diagram. 


Figure  4.  Sample  IDEFO  diagram  showing  box  and  arrow  syntax. 
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box  from  the  bottom  represent  mechanisms.  Mechanisms  are  the 
agents  that  perform  the  activities  represented  in  the  box.  In 
short,  inputs  and  outputs  represent  what  is  done  by  the  process, 
controls  represent  whv  it  is  done,  and  mechanisms  represent  how 
it  is  done. 

The  arrow  structure  of  an  IDEFO  diagram  represents  a 
constraint  relationship  among  boxes.  It  does  not  represent  flow 
of  control  or  sequence.  The  arrows  entering  a  box  show  all  that 
is  needed  by  the  box  to  perform  its  function.  Therefore,  the  box 
is  constrained  by  its  inputs  and  controls. 

Labeling  of  arrows.  Some  arrows  show  both  their  source  and 
destination  boxes  on  the  same  diagram,  while  others  have  one  end 
unconnected  (see  Figure  5) .  The  unconnected  arrows  represent 
inputs,  controls,  or  outputs  of  the  parent  box.  To  find  the 
source  or  destination  of  these  unconnected  arrows,  the  reader 
must  locate  the  matching  arrows  on  the  parent  diagram.  All  such 
unconnected  arrows  must  continue  on  the  parent  for  the  diagrams 
to  be  complete. 

Although  arrow  connections  from  parent  boxes  to  detail 
diagrams  is  sometimes  obvious  from  the  labels,  we  have  developed 
a  special  notation  that  should  allow  readers  to  do  the  match 
quickly.  The  notation  used  to  describe  the  OSBATS  model  is 
slightly  different  from  standard  IDEFO  procedures  for  labeling 
unconnected  arrows.  The  data  for  the  OSBATS  model  are  described 
in  a  structured  data  base.  Each  element  in  the  data  base  is 
identified  by  a  unique  outline  number  (e.g.,  2A1) .  Input  and 
control  arrows  that  represent  data  in  the  data  base  are  labeled 
with  the  appropriate  outline  number.  Often  it  is  the  case  that 
data  are  described  more  generally  at  higher-level  nodes  than  they 
are  at  lower-level  nodes.  Thus,  a  particular  input  or  control 
may  be  labeled  "2"  at  node  AO,  "2A”  at  node  Al,  and  "2A1"  at  node 
A13. 

A  somewhat  different  labeling  scheme  is  used  for  output 
arrows.  Output  arrows  are  labeled  according  to  the  highest-level 
node  at  which  the  output  originated.  For  example,  the  output  of 
node  A212  will  be  labeled  0212A  if  it  does  not  occur  at  any 
higher-level  node.  If  there  are  three  outputs  for  A212,  they 
will  be  labeled  0212A,  0212B,  and  0212C.  The  label  is  consistent 
across  all  nodes  in  which  the  output  is  represented.  Therefore, 
if  the  first  output  for  node  A212  is  also  shown  at  node  A21234, 
it  will  still  be  labeled  0212A.  Occasionally,  the  same  output  is 
represented  at  higher-  and  lower-level  nodes,  but  it  is  more 
detailed  at  the  lower-level  node.  When  this  occurs,  the  output 
will  retain  the  node  number  of  the  higher-level  node  but  will 
receive  an  additional  number  to  represent  the  division  of  the 
output  into  parts.  For  example,  the  output  0212A  may  be 
represented  as  0212A1,  0212A2,  and  0212A3  at  a  lower-level  node. 
If  one  of  these  outputs  is  further  subdivided  at  a  lower-level 
node,  it  will  receive  a  second  letter.  For  example,  if  0212A2  is 
divided  into  three  components,  the  components  will  be  referenced 
as  0212A2A,  0212A2B,  and  0212A2C. 
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Figure  5.  Sample  IDEFO  diagram  showing  source  and  destination. 


Mechanism  arrows  are  used  sparingly  in  the  OSBATS  model 
definition.  When  they  are  used,  their  reference  is  clear. 
Consequently,  the  mechanism  arrows  are  not  numbered  and  are 
identified  only  by  their  label. 

It  is  possible  for  a  data  element  to  serve  as  an  input  to  some 
sub-activities  of  a  given  activity  and  as  a  control  for  other 
sub-activities.  In  this  case,  the  data  will  be  represented  once 
in  the  parent  diagram,  either  as  input  or  control.  In  the 
detailed  diagrams,  the  data  would  be  represented  as  a  control  in 
some  diagrams  and  as  an  input  in  others,  as  appropriate. 

Nodes.  Data,  and  Variables 

This  section  presents  three  tables  that  summarize  information 
about  the  model  nodes,  input  and  control  data,  and  process 
outputs.  Also  included  is  a  list  of  the  symbols  used  to 
represent  the  variables  of  the  model.  Specifically,  the 
following  tables  are  included. 

Table  3  provides  a  list  of  the  nodes  in  the  structure  in  the 
order  that  they  appear  in  the  system  description.  The  node  list 
provides  the  table  of  contents  for  the  IDEFO  model.  If  the  node 
is  represented  by  its  own  detailed  diagram,  the  number  of  that 
diagram  is  listed  in  the  final  column  of  the  node  list.  Nodes 
that  have  no  detailed  diagram  do  not  have  an  IDEF  diagram  number 
listed.  The  descriptions  of  such  a  node  may  be  found  on  the 
diagram  for  its  parent  node. 

Table  4  provides  a  data  base  map  that  enumerates  all  data 
requirements.  The  data  base  map  lists  all  node  numbers  in  which 
each  data  element  data  is  used,  and  specifies  whether  the  data 
are  used  as  an  input  or  a  control  to  each  node. 

Table  5  provides  a  list  of  the  model  outputs  indicating  the 
source  node  and  all  destination  nodes  for  each  output.  A 
destination  listed  as  "UP"  indicates  that  the  output  is 
unconnected  in  a  particular  IDEF  diagram.  The  output  map 
specifies  whether  each  output  is  used  as  an  input  or  control  at 
the  destination  node.  Each  output  description  is  referenced  to  a 
single  IDEF  diagram  number  as  indicated  in  the  output  map. 

Table  6  provides  a  list  of  the  variables  used  in  the  model.  The 
list  provides  a  glossary  of  the  following  three  kinds  of  symbols; 
index  variables,  arguments  of  functions,  and  variables. 
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Table  3.  IDEF  Node  List 


Node 


Title 


IDEF  Nr 


Optimization  of  Simulation-Based 


AO 

A1 

All 

A12 

A13 

A131 

A132 

A133 

A134 

A14 

A141 

A1411 

A1412 

A1413 

A1414 

A142 

A1421 

A1422 

A143 

A1431 

A1432 

A14321 

A14322 

A14323 

A14324 

A1433 


Training  Systems 
Optimize  Training  System  Design 
Perform  Preliminary  Processing 
Analyze  Tasks  for  Instructional  Features 
Analyze  Tasks  for  Fidelity  Features 
Determine  Task  Weights 
Compute  Investment  Allocation 
Compute  Fixed  Operating  Allocation 
Compute  Variable  Operating  Allocation 
Compute  Normalize  Economic  Weights 
Determine  Learning  Function 
Determine  Transfer  Ratio  Parameters 
Determine  Benchmark  Transfer 
Determine  Equation  Exponent 
Determine  Reference  Point 
Determine  Equation  Multiplier 
Determine  Common  Learning  Curve  Parameters 
Determine  Scaling  Factors 
Determine  AE  Headstart 
Determine  Specific  Learning  Curve  Parameter 
Determine  Transfer  Ratios 
Detemnine  Tire  Multiplier 
Determine  Instructional  Feature  Improve 
Determine  AE  Useful  Time  Ratio 
Determine  Device  Useful  Time  Ratio 
Calculate  Time  Multiplier 
Determine  Head  Start 


A2 

A21 

A211 

A2111 

A21111 

A21112 

A211121 

A211122 

A211123 

A21113 

A2112 

A21121 

A21122 

A212 

A2121 

A2122 

A21221 

A21222 

A21223 


Develop  Training  Concept 
Recommend  Simulator  Configuration 
Evaluate  Each  Task 
Evaluate  Simulation  Indicators 
Rate  Simulation  Requirements 
Calculate  Operating  Cost  Savings 
Minimize  Total  Operating  Cost 
Calculate  Cost  Savings 
Calculate  Index  of  OCS 
Calculate  Simulation  Indicators 
Estimate  Development  Costs 
Calculate  Fidelity  Dimension  Costs 
Sum  Across  Dimensions 
Develop  Simulation  Recommendations 
Prepare  Scatterplot  of  Tasks 
Calculate  Simulator  Option  Summaries 
Compute  Cost  Boundary 
Assign  Tasks  to  FMS/PMS/AE  Options 
Sum  Task  Weights  for  FMS/PMS/AE  Options 
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Table  3  (continued) .  IDEF  Node  List 


Node 


A3 

A31 

A311 

A3111 

A3112 

A312 

A3121 

A3122 

A313 

A3131 

A3132 

A3133 

A32 

A321 

A3211 

A3212 

A3213 

A3214 

A322 

A3221 

A3222 

A3223 

A3224 

A323 

A3231 

A3232 

A3233 

A3234 

A4 

A41 

A411 

A412 

A4121 

A4122 

A4123 

A4124 

A4125 

A413 

A4131 

A4132 

A4133 

A4134 


Title 


IDEF  Nr. 


Design  Training  Devices  18 

Select  Instructional  Features  19 

Select  Tasks  and  Candidate  Features  20 

Select  Device  Tasks 
Select  Instructional  Features 
Calculate  Benefits  of  Features  21 

Calculate  Effectiveness  Score 
Calculate  Feature  Benefit 
Select  Optimal  Features  22 

Compute  Benefit/Cost  Ratios  for  Options 
Sort  Options  by  Benefit/Cost  Ratios 
List  Optimal  Features 

Optimize  Device  Fidelity  23 

Construct  Training  Device  Options  24 

Select  Tasks  and  Relevant  Dimensions 


Set  Minimum  and  Maximum  Options 
Incorporate  Instructional  Features  Options 
Define  Intermediate  Options 
Calculate  Costs  and  Benefits  of  Options  25 
Calculate  Costs 
Determine  Task  Trainability 
Calculate  Benefit  Scores 
Calculate  Benefit  Weights 
Compute  Optimal  Device  Designs  26 

Compute  Benefit/Cost  Ratios  for  Options 
Sort  Options  by  Benefit/Cost  Ratios 
Compute  Device  Designs 
Calculate  Cost  Elements 

Assign  Training  to  Devices  27 

Select  Training  Device  for  Tasks  28 

Determine  Training  Device  Hourly  Cost 
Determine  Cost  Effectiveness  Devices  29 

Find  First  Device 
Set  Current  Performance  Level 
Find  Device  Crossover  Points 
Process  Crossover  Points 
Calculate  Training  Time  &  Costs 
Examine  Device  Utilization  30 

Calculate  Total  Device  Hours 
Calculate  Number  of  Simulators  Required 
Determine  Actual  Utilization 
Update  Assumption 
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Table  3  (continued) .  IDEF  Node  List 


Node 

Title  IDEF 

Nr 

A42 

Allocate  Trng  Resources  to  Trng  Devices 

31 

A421 

Detail  Cost  Curves 

32 

A4211 

Detail  Cost  Curves,  Within  Control  Window 

33 

A42111 

Construct  Base  Two-Segment  Cost  Curve 

A42112 

Add  Next  Segment  at  Current  Usage 

A42113 

Add  New  Left  Segment 

A42114 

Add  New  Right  Segment 

A42115 

Check  for  Completion  and  Terminate 

A4212 

Simplify  Using  Control  Window  Shrinkage 

A4213 

Simplify  Using  Detail-Sensitivity  Ordering 

A4222 

Solve  Multi-Task  Resource  Allocation 

Problem 

34 

A4221 

Rank  Cost  Curve  Combinations  by  Minimum 
Potential  Costs 

A4222 

Determine  Multi-Task  Resource  Allocation 
Specific  Cost  Curve  Combination 

35 

A42221 

Conduct  Checks  and  Initializations  for 

Current  Combination 

A42222 

Set  Up  and  Solve  Single-Task  Subproblems 

36 

A422221 

Begin  Next  Algorithm  Pass 

A422222 

Determine  Device  for  Next  Step  in 

Backward  Allocation 

A422223 

Determine  Next  Switching  Performance 

Level 

A422224 

Allocate  Resources  to  Step,  Forcing 

Lumping  from  Previous  Pass 

A422225 

Determine  Next  Algorithm  Action 

A4223 

Characterize  Solution  and  Adjust  Surcharges 

To  Enforce  Constraint  Compliance 

37 

A42231 

Check  for  Constraint  Violation  and 

Adjust  Surcharges 

A42232 

Check  for  Satisfactory  Convergence  and 

Check  for  Optimality  of  Combination 
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Table  4.  Data  Base  Map 


DB  Description 
Cat. 


1  Task  Training  Requirements 


lA  Task  Learning  Points 


lAl  Entry  Performance  Level 


1A2  Performance  Standard 


IB  Simulation  Requirements 


Destination 

Input  or 
Control 

IDEF 

Number 

AO 

Control 

OPT-1 

A1 

Control 

OPT-2 

A2 

Control 

OPT- 2 

A3 

Control 

OPT-2 

A4 

Control 

OPT-2 

A21 

Control 

OPT- 10 

A31 

Control 

OPT-18 

A311 

Control 

OPT- 19 

A3111 

Control 

OPT-2 0 

A41 

Control 

OPT-27 

A42 

Control 

OPT-27 

All 

Input 

OPT- 3 

A14 

Input 

OPT-3 

A142 

Input 

OPT-5 

A211 

Control 

OPT- 11 

A2111 

Control 

OPT- 12 

A21112 

Control 

OPT-13 

A211121 

Control 

OPT- 14 

A1421 

Input 

OPT-7 

A1422 

Input 

OPT- 7 

A1433 

Input 

OPT-8 

A412 

Control 

OPT-2 8 

A4121 

Control 

OPT-29 

A4122 

Control 

OPT-2 9 

A422 

Control 

OPT- 31 

A4222 

Control 

OPT-34 

A42222 

Control 

OPT- 3 5 

A422222 

Control 

OPT-36 

A422223 

Control 

OPT-3 6 

A422224 

Control 

OPT-36 

A141 

Input 

OPT-5 

A1411 

Input 

OPT- 6 

A1421 

Input 

OPT-7 

A412 

Control 

OPT-2 8 

A4124 

Control 

OPT-29 

A422 

Control 

OPT-3 1 

A4222 

Control 

OPT-34 

A42222 

Control 

OPT-35 

A422221 

Control 

OPT-36 

A211 

Control 

OPT- 11 

A2111 

Control 

OPT-12 

A21111 

Control 

OPT-13 
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Table  4  (continued) .  Data  Base  Map 


DB 

Cat. 

Description 

Destination 

Input  or 
Control 

IDEF 

Number 

2 

Other  Task  Data 

AO 

Input 

OPT-1 

A1 

Input 

OPT-2 

A2 

Input 

OPT-2 

2A 

Task  Training  Hours  and  Costs 

A13 

Input 

OPT- 3 

A14 

Input 

OPT- 3 

A131 

Input 

OPT-4 

A133 

Input 

OPT-4 

A142 

Input 

OPT- 5 

A143 

Input 

OPT-5 

A1432 

Input 

OPT-8 

A14322 

Input 

OPT-9 

A14323 

Input 

OPT-9 

A21 

Input 

OPT- 10 

A211 

Input 

OPT- 11 

A2111 

Input 

OPT-12 

A21112 

Input 

OPT-13 

A211121 

Input 

OPT-14 

2A5 

Other  training  costs 

A123 

Input 

OPT-4 

2B 

Task  Information  Processing 

All 

Input 

OPT-3 

Characteristics 

A12 

Input 

OPT-3 

2C 

Task  Activities 

A12 

Input 

OPT-3 

3 

Training  Device  Data 

AO 

Input 

OPT-1 

A1 

Input 

OPT-2 

A2 

Input 

OPT-2 

A4 

Input 

OPT-2 

A21 

Input 

OPT-10 

A211 

Input 

OPT- 11 

A212 

Input 

OPT- 11 

A42 

Input 

OPT-27 

3A 

Training  Device  Costs 

A13 

Input 

OPT-3 

A2111 

Input 

OPT-13 

A21112 

Input 

OPT-13 

A211121 

Input 

OPT-14 

A211122 

Input 

OPT-14 

A41 

Input 

OPT-27 

A411 

Input 

OPT-28 

A421 

Input 

OPT-3 1 

A4211 

Input 

OPT- 3 2 

A422 

Input 

OPT- 3 2 

A42111 

Input 

OPT-3 3 

3A1 

Investment  Cost 

A131 

Input 

OPT-4 
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Table  4  (continued) .  Data  Base  Map 


DB 

Cat. 

Description 

Destination  Input  or 
Control 

IDEF 

Number 

3A2 

Annual  Fixed  Operating  Cost 

A132 

Input 

OPT-4 

3A3 

Hourly  Variable  Operating  Cost 

A133 

Input 

OPT-4 

3A4 

Lifecycle 

A13 

Control 

OPT- 3 

A131 

Control 

OPT-4 

A2122 

Input 

OPT- 16 

A21221 

Input 

OPT-17 

3A5 

Maximum  Annual  Utilization 

A131 

Input 

OPT-4 

A4132 

Control 

OPT-30 

3B 

Training  Device  Cue  &  Response 

A14 

Input 

OPT-3 

Capabilities 

A143 

Input 

OPT-5 

A1431 

Input 

OPT-8 

3C 

Training-Device  Instructional 

A14 

Input 

OPT-3 

Features 

A143 

Input 

OPT-5 

A1432 

Input 

OPT-8 

A14321 

Input 

OPT-9 

4 

Fidelity  Dimension  Data 

AO 

Input 

OPT-1 

A2 

Input 

OPT-2 

A3 

Input 

OPT-2 

A21 

Input 

OPT- 10 

A211 

Input 

OPT- 11 

A32 

Input 

OPT-18 

4A 

Fidelity  Dimensions  and 

A2112 

Input 

OPT-12 

Levels 

A21121 

Input 

OPT- 15 

A321 

Input 

OPT-2 3 

A3211 

Input 

OPT-24 

A3212 

Input 

OPT-24 

A3214 

Input 

OPT-24 

4B 

Fidelity  Dimension  Cost  Data 

A2112 

Input 

OPT-12 

A21121 

Input 

OPT-15 

A322 

Input 

OPT-2 3 

A3221 

Input 

OPT-2 5 

4C 

Minimum  Performance  Parameter 

A14 

Input 

OPT-3 

A141 

Input 

OPT-5 

A1412 

Input 

OPT- 6 

A1413 

Input 

OPT-6 

4D 

Fidelity  Rules 

AO 

Control 

OPT-1 

A1 

Control 

OPT-2 

A12 

Control 

OPT-3 
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Table  4  (continued) .  Data  Base  Map 


DB 

Cat. 

Description 

Destination 

Input  or 
Control 

IDEF 

Number 

5A 

Instructional  Feature  Rules 

AO 

Control 

OPT-1 

A1 

Control 

OPT-2 

All 

Control 

OPT-3 

5B 

Instructional  Feature  Cost  and 

AO 

Input 

OPT-1 

Weight 

A3 

Input 

OPT-2 

A31 

Input 

OPT-18 

A311 

Input 

OPT- 19 

A3112 

Input 

OPT-2 0 

5B1 

Instructional  Feature  Cost 

A313 

Input 

OPT-19 

A3131 

Input 

OPT-22 

5B2 

Instructional  Frature  Benefit 

A312 

Input 

OPT-19 

Weight 

A3122 

Input 

OPT-2 1 

6 

Training  System  Data 

AO 

Control 

OPT-1 

A1 

Control 

OPT-2 

A2 

Control 

OPT-2 

A3 

Control 

OPT-2 

A4 

Control 

OPT-2 

A22 

Control 

OPT-10 

A211 

Control 

OPT- 11 

A212 

Control 

OPT- 11 

A2111 

Control 

OPT-12 

A21113 

Control 

OPT-13 

A2122 

Control 

OPT-16 

A41 

Control 

OPT-27 

A42 

Control 

OPT-27 

A413 

Control 

OPT-28 

6A1 

Annual  Student  Throughput 

A13 

Control 

OPT-3 

A131 

Control 

OPT-4 

A4132 

Control 

OPT- 30 

A4133 

Control 

OPT- 30 

6B 

Model  Information 

A14 

Control 

OPT-3 

A143 

Control 

OPT-5 

A1432 

Control 

OPT-8 

A32 

Control 

OPT-18 

A323 

Control 

OPT-2 3 

A3234 

Control 

OPT-26 

6B1 

Standard  Adjustment  Rate 

A141 

Control 

OPT-5 

A1411 

Control 

OPT-6 
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Table  4  (continued) .  Data  Base  Map 


DB 

Cat. 

Description 

Destination 

Input  or 
Control 

IDEF 

Number 

6B2 

Learning  Curve  Exponent 

A142 

Control 

OPT-5 

A143 

Control 

OPT-5 

A1421 

Control 

OPT- 7 

A1422 

Control 

OPT-7 

A1433 

Control 

OPT-8 

6B3 

Maximum  Instructional 

Feature 

A14321 

Control 

OPT-9 

Effect 

6B5 

Assumed  Setup  Savings 

Percentage 

A13323 

Control 

OPT-9 

6B6 

Cost  Savings  Weight 

A21221 

Control 

OPT- 17 

6B7 

Recommendation  Boundaries 

A21222 

Control 

OPT-17 
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Table  5.  IDEF  Output  Map 


Output 

Cat. 

Description 

Source 

Destin¬ 

ation 

Input  or 
Control 

IDEF 

Num. 

OlA 

Instructional  Feature 

A1 

A3 

Input 

2 

Appropriateness 

All 

A14 

Control 

3 

All 

UP 

3 

OIB 

Task  Cue  &  Response 

A1 

A2 

Control 

2 

Requirements 

A1 

A3 

Control 

2 

A12 

A14 

Control 

3 

A12 

UP 

3 

OIC 

Task  Economic  Weights 

A1 

A2 

Input 

2 

A1 

A3 

Input 

2 

A13 

UP 

3 

A134 

UP 

4 

OID 

Learning  Function 

A1 

A2 

Input 

2 

Parameters 

A1 

A4 

Input 

2 

A14 

UP 

3 

OlDl 

Time  Scaling  Factor 

A142 

UP 

5 

A1421 

A1422 

Control 

7 

A1421 

UP 

7 

01D2 

AE  Head  Start 

A142 

A143 

Control 

5 

A142 

UP 

5 

A1422 

UP 

7 

01D3 

Device  Specific 

A143 

UP 

5 

Learning  Parameters 

01D3A 

Learning  Curve 

A1431 

A1433 

Control 

8 

Asymptotes 

A1431 

UP 

8 

01D3B 

Time  Multiplier 

A1432 

A1433 

Control 

8 

A1432 

UP 

8 

A14324 

UP 

9 

01D3C 

Head  Start 

A1433 

UP 

8 

OlE 

Baseline  Variable 

A1 

A2 

Input 

2 

Operating  Costs 

A13 

UP 

3 

A133 

UP 

4 

A133 

A134 

Input 

4 

013A 

Total  AE  Training  Time 

A13 

A14 

Input 

3 

A133 

UP 

4 

0131A 

CINV 

A131 

A134 

Input 

4 

0131B 

TAE 

A131 

A133 

Input 

4 
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Table  5.  (continued) .  IDEF  Output  Map 


Output 

Cat. 

Description 

Source 

Destin¬ 

ation 

Input  or 
Control 

IDEF 

Num. 

0131C 

NAE  &  TAENORM 

A131 

A132 

Input 

4 

0132A 

CFX 

A132 

A134 

Input 

4 

0141A 

Intermediate  Parameters 

A141 

A143 

Control 

5 

0141A1 

AR 

A1411 

A1412 

Control 

6 

A1411 

UP 

6 

0141A2 

Q 

A1412 

A1413 

Control 

6 

A1412 

UP 

6 

014 1A3 

b 

A1413 

A1414 

Control 

6 

A1413 

UP 

6 

014 1A4 

g 

A1414 

UP 

6 

014321A 

lADJ  T,i 

A14321 

A14324 

Input 

9 

014322A 

U  T,AE 

A14322 

A14324 

Input 

9 

014323A 

U  T,i 

A14323 

A14324 

Input 

9 

02A 

Simulation 

A2 

A3 

Control 

2 

Recommendation 

A2 

UP 

2 

A21 

UP 

10 

A212 

UP 

11 

02A1 

Task  Assignments  to 

A2122 

UP 

16 

Options 

A21222 

A21223 

Control 

17 

A21222 

UP 

17 

02A2 

Option  Summaries 

A2122 

UP 

16 

A21223 

UP 

17 

0211A 

Simulation  Requirements 

A211 

A212 

Input 

11 

Indicators 

A2111 

UP 

12 

A21113 

UP 

13 

0211B 

Development  Cost 

A211 

A212 

Input 

11 

Estimates 

A2112 

UP 

12 

A21122 

UP 

15 

021111A 

Simulation  Requirements 

A21111 

A21113 

Input 

13 

Indices 

021112A 

Operating  Cost  Savings 

A21112 

UP 

13 

A211122 

A211123 

Input 

14 

A211122 

UP 

14 
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Table  5.  (continued).  IDEF  Output  Map 


Output 

Cat. 

Description 

Source 

Destin- 

dtion 

Input  or 
Control 

IDEF 

Num. 

021112B 

Operating  Cost  Savings 

A21112 

A21113 

Input 

13 

Indices 

A211123 

UP 

14 

0211121A 

AE  Hours  Savings 

A211121 

A211122 

Control 

14 

A211121 

A211123 

Control 

14 

0211121B 

FMS  Training  Time 

A211121 

A211122 

Control 

14 

021121A 

Required  Cue  &  Response 

A21121 

A21122 

Input 

15 

Cost 

02121A 

Scatterplot 

A2121 

A2122 

Control 

16 

021221A 

AE  Cost  Boundary 

A21221 

A21222 

Control 

17 

03A 

Optimal  Device  Designs 

A3 

UP 

2 

03A1 

Selected  Instructional 

A31 

A32 

Input 

18 

Features 

A31 

UP 

18 

A313 

UP 

19 

A3133 

UP 

22 

03A2 

Optimal  Device 

A3 

A1 

Input 

2 

Capability 

A32 

UP 

18 

A323 

UP 

23 

A3233 

UP 

26 

03A3 

Device  Cost  Elements 

A3 

A4 

Input 

2 

A32 

UP 

18 

A323 

UP 

23 

A3234 

UP 

26 

0311A 

Candidate  Instructional 

A311 

A312 

Control 

19 

Features 

A311 

A313 

Control 

19 

A3112 

UP 

20 

0311B 

Selected  Tasks 

A311 

A312 

Control 

19 

A3111 

A3112 

Control 

20 

A3111 

UP 

20 

0312A 

Feature  Benefit 

A312 

A313 

Input 

19 

A3122 

UP 

21 

03121A 

Weighted  Feature 

A3121 

A3122 

Input 

21 

Effectiveness 

03131A 

Benefit/Cost  Ratio  for 

A3131 

A3132 

Input 

22 

All  Options 
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Table  5. 

(continued) .  IDEF  Output 

Map 

Output 

Description 

Source 

Destin¬ 

Input  or 

IDEF 

Cat. 

ation 

Control 

Num. 

03132A 

Benefit/Cost  Priority 
of  Options 

A3132 

A3133 

Input 

22 

032  lA 

Training  Device  Options 

A321 

A322 

Control 

23 

A321 

A323 

Control 

23 

A3214 

UP 

24 

0321B 

Technical  Performance 

A321 

A322 

Input 

23 

Indices 

A3214 

UP 

24 

03211A 

Relevant  Tasks 

A3211 

A3212 

Control 

24 

03211B 

Relevant  Dimensions 

A3211 

A3212 

Input 

24 

A3211 

A3214 

Input 

24 

03212A 

Minimum  and  Maximum 
Options 

A3212 

A3214 

Control 

24 

03213A 

Instructional  Feature 
Options 

A3213 

A3214 

Control 

24 

0322A 

Costs  and  Benefits  of 
Options 

A322 

A323 

Input 

23 

0322A1 

Option  Costs 

A3221 

UP 

25 

0322A2 

Option  Benefit  Scores 

A3223 

A3224 

Input 

25 

A3223 

UP 

25 

0322A3 

Fidelity  Dimension 
Weights 

A3224 

UP 

25 

03222A 

Tasks  Trained  by  Option 

A3222 

A3223 

Input 

25 

03231A 

Benefit/Cost  Ratios  For 
Options 

A3231 

A3232 

Input 

26 

03232A 

Benefit/Cost  Priority 
of  Options 

A3232 

A3233 

Input 

26 

03233A 

Selected  Design 
Development  Cost 

A3233 

A3234 

Input 

26 

04A 

Efficient  Devices 

A4 

A3 

Control 

2 

A4 

UP 

2 

A41 

A42 

Input 

27 

A41 

UP 

27 
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Table  5.  (continued) .  IDEF  Output  Map 


Output 

Cat. 

Description 

Source 

Destin¬ 

ation 

Input  or 
Control 

IDEF 

Num. 

04A1 

Selected  Devices 

A412 

UP 

28 

A4121 

A4123 

Input 

29 

A4121 

UP 

29 

A4124 

UP 

29 

04A2 

Costs 

A412 

UP 

28 

A4125 

UP 

29 

04A3 

Hours/Task 

A412 

A413 

Input 

28 

A412 

UP 

28 

A4125 

UP 

29 

04B 

Optimal  Allocation 

A4 

UP 

2 

A42 

UP 

27 

A423 

UP 

31 

0411A 

Device  Hourly  Cost 

A411 

A412 

Control 

28 

0411B 

Currently  Assumed 

A411 

A413 

Input 

28 

Utilization 

04122A 

Current  Performance 

A4122 

A4123 

Input 

29 

Level 

A4122 

A4124 

Control 

29 

A4122 

A4125 

Input 

29 

04123A 

Crossover  Points 

A4123 

A4124 

Control 

29 

04124A 

Device  Performance 

A4124 

A4122 

Input 

29 

Criterion 

A4124 

A4125 

Input 

29 

0413A 

Revised  Device 

A413 

A411 

Input 

28 

Utilization 

A4134 

UP 

30 

04131A 

Total  Hours  by  Device 

A4131 

A4132 

Input 

30 

A4131 

A4133 

Input 

30 

04132A 

Required  Training 

A4132 

A4133 

Control 

30 

Devices 

04133A 

Actual  Utilization 

A4133 

A4134 

Control 

30 

0421A 

Number  of  Combinations 

A421 

A422 

Input 

31 

0421B 

Cost  Curve  Parameters 

A421 

A422 

Input 

31 

A421 

A423 

Input 

31 

A4213 

UP 

32 

04211A 

Candidate  Cost  Curve 

A4211 

A4212 

Input 

32 

Parameters 

A42115 

UP 

33 
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Table  5.  (continued) .  IDEF  Output  Map 


Output 

Description 

Source 

Destin¬ 

Input  or 

IDEF 

Cat. 

ation 

Control 

Num. 

042111A 

Interim  Cost  Curve 
Parameter 

A42111 

A42112 

Input 

33 

042112A 

Interim  Cost  Curve 
Parameter 

A42112 

A42113 

Input 

33 

042113A 

Interim  Cost  Curve 
Parameter 

A42113 

A42J.14 

Input 

33 

042113B 

NEWL,  NEWR,  NSEGL,  NSEGR 

A42113 

A42113 

Control 

33 

042114A 

Interim  Cost  Curve 
Parameter 

A42114 

A42115 

Input 

33 

042114B 

NEWL,  NEWR,  NSEGL,  NSEGR 

A42114 

A42113 

Control 

33 

042115A 

Program  Control 

A42115 

A42113 

Control 

33 

042115B 

NEWL,  NEWR,  NSEGL,  NSEGR 

A42115 

A42113 

Control 

33 

04212A 

Revised  Control  Window 
Parameter 

A4212 

A4211 

Control 

32 

04212B 

Revised  Cost  Curve 

A4212 

A4213 

Input 

32 

Parameters 

A4212 

UP 

32 

04213A 

Identity  of  Least 

Detail  Sensitive  Device 

A4213 

A4211 

Control 

32 

0422A 

Current  Resource 

A422 

A423 

Control 

31 

Allocation  Solution 

A4223 

A4222 

Control 

34 

A4223 

UP 

34 

0422A1 

Incumbent  Best  Combo 
and  Solution 

A42232 

UP 

37 

04221A 

Cost  Combo  Ranking 

A4221 

A4222 

Control 

34 

04221B 

Minimum  Potential  Costs 

A4221 

A4222 

Input 

34 

04222A 

Tentative  Resource 

A4222 

A4223 

Input 

34 

Allocation  Solution 

A42222 

UP 

35 

04222A1 

Qo]  r)e»\rHr'«a 

Training  Sequence 

A422225 

UP 

36 

04222A2 

Selected  Device 

Resource  Allocation 

A422225 

UP 

36 

042221A 

Initial  Surcharges 

A42221 

A42222 

Input 

35 
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Table  5.  (continued).  IDEF  Output  Map 


Output 

Description 

Source 

Destin- 

Input  or 

IDEF 

Cat. 

ation 

Control 

Num. 

042221B 

Resource  Allocation 

Upper  Bounds 

A42221 

A42222 

Input 

35 

042222A 

Training  Task  Counter 

A42222 

A42222 

Control 

35 

A422225 

A422221 

Control 

36 

0422221A 

Initial  Number  of  Stages 

A422221 

A422222 

Input 

36 

0422221B 

Current  Performance 

Level 

A422221 

A422222 

Input 

36 

0422221C 

Effective  Device  Cost 

A422221 

A422222 

Input 

36 

Rates 

A422221 

A422223 

Input 

36 

0422222A 

Incumbent  Device 

A422222 

A422223 

Input 

36 

0422222B 

Current  Performance 

Level 

A422222 

A422223 

Input 

36 

0422223A 

Next  Device  or  No-Next 
Device 

A422223 

A422225 

Input 

36 

0422223B 

Next  Performance  Level 

A422223 

A422224 

Input 

36 

0422224A 

Number  of  Stages 

A422224 

A422225 

Input 

36 

0422224B 

Multiple  Allocation 

Flags 

A422224 

A422225 

Input 

36 

0422224C 

Current  Performance 

Level 

A422224 

A422225 

Input 

36 

0422224D 

Aggregate  Allocation  to 
Incumbent  Device 

A422224 

A422225 

Input 

36 

0422225A 

Pass 

A422225 

A422221 

Control 

36 

0422225B 

Incumbent  Device 

A422225 

A422223 

Input 

36 

0422225C 

Current  Performance 

A422225 

A422222 

Input 

36 

Level 

A422225 

A422223 

Input 

36 

04223A 

Iteration  Counter 

A4223 

A4222 

Control 

34 

A42232 

UP 

37 

04223B 

Surcharges 

A4223 

A4222 

Control 

34 

A42231 

A42232 

Input 

37 

A42231 

UP 

37 
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Table  5.  (continued) .  IDEF  Output  Map 


Output 

Cat. 

Description 

Source 

Destin¬ 

ation 

Input  or 
Control 

IDEF 

Num. 

04223C 

Cost  Curve  Combo 

A4223 

A4222 

Control 

34 

A42232 

UP 

37 

0423A 

Iteration  and  Flag 
Counter 

A423 

A421 

Control 

31 

0423B 

Last  Resource 

Solution 

A423 

A421 

Control 

31 

64 


Table  6.  OsBATS  Variable  List 


Indices; 


AE 

FMS 

i 

INCMB 

INEXT 

j 

k 

KB 

KBEST 

KOMBO 

m 

M 

n 

R 

s 

T 


Actual  equipment  as  a  training  device 

Full  mission  simulator 

Index  of  training  devices 

Index  of  the  incumbent  training  device 

Index  of  the  next  device  to  be  used  for  training 

Index  of  cue/response  dimensions 

Index  of  instructional  features 

Index  of  ranks 

Index  of  best  solution 

Index  of  cost-curve  combinations 

Index  of  fidelity  dimension  options 

Index  of  the  most  capable  fidelity  dimension 

option 

Index  of  number  of  devices  acquired  for  cost  curve 
segments 

Hypothetical  training  device  that  exactly  meets 
all  cue  and  response  requirements 
Selected  device 
Index  of  tasks 


Arguments  of  Functions; 


t  =  Time 

TPj^  =  Absolute  technical  performance  index  for  training 
device  option  m  on  fidelity  dimension  j 

Variables; 


Ain 

ALLOCM 


ASM 

device  R 

ASM 


TR 


ASSETS; 


B; 


BEN^ 

BWTj. 

(B/C)j, 


=  Height  of  cost  curve  CUMCj(t)  at  beginning  of 
segment  n 

=  Flag  to  indicate  that  multiple  allocations  have 
occurred  on  this  pass 

=  Benchmark  asymptote  for  task  T  on  hypothetical 

=  Asymptote  of  learning  curve  for  task  T  on  device  i 
=  Number  of  training  devices  of  type  i  already  owned 
and  available  for  use  without  investment  cost 
= Current  effective  price  of  device  i  usage,  equal 
to  the  product  of  the  current  cost  curve  segment 
slope,  B.^,  and  the  current  device  i  surcharge, 
SURCHGj 

=  Slope  of  segment  n  of  the  cost  curve  CUMCj{t) 

=  Cue  and  response  reference  point  for  task  T  and 
fidelity  dimension  j 
= Benefit  of  instructional  feature  k 
= Benefit  weight  for  simulator  options  on 
cue/response  dimension  j 

=  Incremental,  weighted  benefit-to-cost  ratio  of 
simulator  option  m  on  cue/response  dimension  j 
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Table  6.  OSBATS  Variable  List 


C^j  =  the  cost  of  using  device  i  on  task  T,  as 

calculated  by  Resource  Allocation  Module  single 
task  optimization 

CBEST  =  The  per-student  jost  of  training  under  the 
incumbent  best  solution 

CDEVj^(TPj^)  =  Development  cost  of  simulator  option  m  with 
capability  (TPj^)  on  cue/response  dimension  j 
CFXy  = Fixed  cost  per  year  allocated  to  the  training  of 
task  T 

CINV^  = Investment  cost  per  year  allocated  to  the  training 
of  task  T 

CLB,(Qj,gg  =  The  lower  bound  on  possible  per-student  cost  is 
found  for  combination  KOMBO 
CLEFT  =  The  minimum  cost  in  the  range  of  detailing 
CMAXj  =  Development  cost  of  the  maximum  technical 

performance  index  (set  to  1.0)  on  cue/response 
dimension  j 

CMINj  =  Development  cost  of  the  minimum  technical 
performance  index  (not  necessarily  0.0)  on 
cue/response  dimension  j 

CONDTN  =  Flag  that  indicates  wt -  her  resource  allocation 
solution  has  been  founc. 

GRANGE  =  The  range  within  which  a  cost  curve  must  be 
detailed,  expressed  in  dollars  per  student 

CRELj^(TPj^)  =  Normalized  development  cost  of  simulator  option  m 
with  capability  (TPj^)  on  fidelity  dimension  j 
CRH^  =  Number  of  classroom  hours  needed  to  train  task  T 
in  conjunction  with  the  actual  equipment  from 
entry  to  standard  without  use  of  a  simulator 
CRIGHT  =  The  maximum  cost  in  the  range  of  detailing 
CUMC;(t)  = Cumulative  cost  of  training  on  device  i  as  a 
function  of  per  student  training  time 
CVARy  = Variable  cost  per  year  allocated  to  the  training 
of  task  T 

DCST^j  (FRQT^j)  =  Development  cost  needed  to  achieve  the 

cue/response  requirement  of  task  T  on  dimension  j 
DEVCy  =  Development  cost  estimate  for  a  simulator  to  train 
task  T 

DEVTj  = Total  training  time  across  all  tasks  on  device  i 
DVIFjjj  =  Presence  of  instructional  feature  k  on  device  i 
Ej  = Empirically  determined  parameter  in  a  development 
cost  equation  for  cue/response  dimension  j 
EFF,^  =  Aggregation  of  effectiveness  over  chosen  tasks  for 
instructional  feature  k 
ENT^  =  Entry  level  performance  for  task  T 
EPMIN^j  =  User-specified  minimum  entry  performance  for  task 
T  on  device  i 

ERRl  =  The  maximum  error  that  can  occur  if  a  cost  curve 
curve  is  simplified  to  a  single  infinite  segment 
ERR2  =  The  maximum  error  that  can  occur  if  a  cost  curve 


is  simplified  to  exactly  two  segments 
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Table  6.  OSBATS  Variable  List 


FCA  = 
FCAMAX  = 

FCAPjj  = 

FCFAC  = 
FMINj  = 

FOC,  = 
FRQT^j  = 

GRADS  = 
HSt,-  = 
lADJ^^  = 

IDEVC^  = 

IDV,^  = 
IFBN  = 

IFq  = 
= 

IFT,,  = 
IFWT^  = 

INV.  = 
lOCS^  = 

ISIM^  = 

ISR^  = 

JC^  = 

ki  = 

KALLOC,  = 

KASE  = 

KBRANK(KB)  = 

LC,  = 
LCEST  = 


The  set  of  instructional  features  included  in 
level  m  of  instructional  features  dimension  in 
Fidelity  Optimization  Module 

Fractional  change  allowed  in  change  of  SURCHGj 
Initial  setting  of  FCA 

Cue  and  response  capability  on  dimension  j  on 
device  i 

Factor  for  estimating  annual  fixed  cost  for 
training  device  design  from  investment  cost 
Minimum  performance  when  capability  on  fidelity 
dimension  j  is  null  but  capabilities  on  all  other 
dimensions  are  perfect 

Fixed  operating  cost  per  year  of  device  i 
Cue  and  response  requirement  for  task  T  on 
fidelity  dimension  j 

Required  number  of  graduates  per  year 
Head  start  for  task  T  on  device  i 
Instructional  feature  improvement  in  efficiency 
for  task  T  on  device  i 

Development  cost  index  for  a  simulator  to  train 
task  T 

A  sequence  of  training  devices 
Maximum  percentage  improvement  in  training 
efficiency  for  instructional  features 
Development  cost  for  instructional  feature  k 
Number  of  appropriate  instructional  features  for 
task  T  on  device  i 

Instructional  features  task  match  for  task  T  and 
instructional  feature  k 

Benefit  weight  for  instructional  feature  k  that 
includes  frequency  of  use,  instructor  loading  and 
useability 

Investment  cost  of  one  device  i 

Index  of  operations  cost  savings  due  to  a 

simulator  for  task  T 

Indicator  that  a  simulator  is  appropriate  for 
training  task  T 

Index  for  the  importance  of  simulator  requirements 
for  training  task  T 

The  number  of  fidelity  dimensions,  j,  such  that 
FRQT^J  >  0  and  FMINj  >  0  for  task  T 
The  cost  curve  segment  being  used  for  device  i 
under  combination  KOMBO 

Counter  to  keep  track  of  the  number  of  times 
device  I  has  received  allocations  on  a  pass 
A  counter  that  determines  whether  costs  are  to  be 
detailed 

the  cost-curve  combination  ranked  KB  according  to 
the  minimum  cost 

Life  cycle  (in  years)  of  device  i 

Estimate  for  life  cycle  (in  years)  of  device 

designs  produced  by  Fidelity  Optimization  Module 
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Table  6.  OSBATS  Variable  List 


LEV,.  = 

MAXIF  = 
MAXKB  = 

Ni  = 
NAE  = 


NBEN.  = 

jm 

NCOMBO  = 

ND  = 

NDEV.  = 

NEW  = 
NEWL  = 
NEWR  = 
NOHj  = 


NRANGE  = 
NSEGL  = 

NSEGR  = 

NSTAGE  = 
NTABj  = 

OCS^  = 

OH^  = 


OTC^  = 
PTi(t)  = 

Pjk  = 

P  = 
^MINI 

P  = 

•^next 

P  = 

^NOW 

PASS  = 

PB  = 
PENTRY  = 


Identification  of  current  segment  for  cost  curve 
for  device  i 

Maximum  number  of  useful  instructional  features 
the  maximum  allowable  number  of  cost-curve 
combinations 

Number  of  segments  in  the  cost  curve,  CUMCj(t) 
Number  of  actual  equipment  devices  needed  to  train 
the  requisite  graduates  per  year  without  the  use 
of  simulators 

Normalized  benefit  score  of  simulator  option  m  on 
cue/response  dimension  j 

The  total  number  of  combinations  of  cost-curve 
segments 

The  number  of  training  devices  under  consideration 
by  the  Resource  Allocation  Module 

Number  of  device  i's  needed  to  train  the  requisite 
number  of  graduates  per  year 

The  number  of  new  devices  associated  required 

The  number  of  new  devices  associated  with  NSEGL 

The  number  of  new  devices  associated  with  NSEGR 

Number  of  non-operational  hours  on  the  actual 
equipment  in  order  to  train  task  T  from  entry  to 
standard  without  use  of  a  simulator 
The  number  of  segments  desired  in  a  cost  curve 
The  leftmost  deliberately  detailed  segment  of  the 
cost  curve 

The  rightmost  deliberately  detailed  segment  of  the 
cost  curve 

Counter  of  devices  selected  for  task 

The  number  of  segments  in  the  cost  curve  for 

device  i 

Operations-cost  savings  for  task  T  that  result 
from  using  a  FMS 

Number  of  operational  hours  on  the  actual 
equipment  in  order  to  train  task  T  from  entry  to 
standard  without  use  of  a  simulator 
Other  costs  of  training  task  T,  such  as  ammunition 
Performance  on  task  T  with  device  i  as  a  function 
of  time 

Performance  at  completion  of  k-th  stage  training, 
on  device  j,^ 

The  minimum  performance  level  that  can  be  reached 
using  the  incumbent  device 

Transition  performance  at  which  device  INEXT  will 
be  used 

Current  performance  level 

Counter  that  reflects  the  current  pass  through  the 
resource  allocation  problem 

Payback  period  needed  for  a  full  mission  simulator 
Performance  level  at  beginning  of  training  on  a 
device 
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Table  6.  OSBATS  Variable  List 


PERFj(TRNTj,P^„J 


PEXIT  = 

PINV.(p)  = 
PRATE.  = 


PX 


si 


= 

Qt  = 
r  = 

RGRT21,-  = 


RGRT31j  = 


RGRT32,  = 


RM 


Tj 


SADJ  = 
STD^  = 
STEPHT.  = 

SUH^  = 
SURCHGj  = 
SUSAV  = 
Tn  - 

TAE^  = 


TAENRM^  = 
TAVAIL.  = 
TCST^  = 


TLEFT  - 
TLRNC  = 


=  Performance  achieved  on  training  device  i 

after  training  an  amount  of  time  TRNT,-,  given 
an  entry  j.evel  of  performance  P^ou* 

Performance  level  at  conclusion  of  training  on  a 
device 

The  inverse  function  of  PERF,- 

Rate  of  increase  in  performance  with  respect  to 
per-student  dollar  spent  at  that  performance 
level . 

Crossover  performance  level  between  the  selected 
device  and  device  i 

Multiplier  of  asymptote  equation  for  task  T 
Exponent  of  asymptote  equation  for  task  T 
learning  curve  exponent 

The  increase  in  possible  cost  error  for  device  i 
if  a  cost  curve  of  a  single  infinite  segment  is 
used  instead  of  a  curve  with  two  segments;  i.e. 
the  potential  regret. 

The  increase  in  possible  cost  error  for  device  i 
if  a  cost  curve  of  a  single  infinite  segment  is 
used  instead  of  a  curve  with  three  or  more 
segments;  i.e.  the  potential  regret. 

The  increase  in  possible  cost  error  for  device  i 
if  a  cost  curve  of  a  two  segments  is  used  instead 
of  a  curve  with  three  or  more  segments;  i.e.  the 
potential  regret. 

Modified  task  cue  and  response  requirement  for 
task  T  and  cue/response  dimension  j ,  such  that  it 
is  zero  for  least  capable  simulator  option  on 
dimension  j  and  one  for  the  most  capable  option 
Adjustment  %  for  benchmark  transfer 
Training  standard  for  task  T 

the  size  of  the  step  between  adjacent  segments  of 

a  detailed  cost  curve,  for  device  i 

Setup  hours  on  the  actual  equipment  for  task  T 

The  cost  surcharge  for  each  device  i 

Setup  time  savings 

Starting  value  of  t  for  segment  n  of  cost  curve, 
CUMC.  (t) 

Total  number  of  training  hours  on  the  actual 
equipment  (not  including  the  classroom)  in  order 
to  train  task  T  from  entry  to  standard  without  use 
of  a  simulator 

Normalized  value  of  TAE^  for  task  T 
Current  per-student  device  i  time  constraint 
Total  life  cycle  training  cost  that  can  be 
allocated  to  task  T 

Starting  value  (per-student  time,  t)  for  h-th 
segment  of  detailed  cost  curve  CUMC^  (t) 

The  lowest  time  in  the  in  the  domain  of  detailing 
Tolerance  parameter 

Time  multiplication  factor  for  task  T  on  device  i 
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Table  6.  OSBATS  Variable  List 


TPMAX;  = 


TPMIN;  = 


TPREL.JTPjJ 


TRANCE  = 


TRIGHT 

TRINC 


TRNT^. 


TSIM  = 


TTOTt  = 


Absolute  technical  performance  index  for  training 
device  option  m  on  fidelity  dimension  j 
Technical  performance  of  the  most  capable  FMS 
option  for  cue/response  dimension  j 
Technical  performance  of  the  least  capable  FMS 
option  for  cue/response  dimension  j 
Normalized  technical  performance  of  simulator 
option  m  with  capability  TPj^  on  cue/response 
dimension  j 

The  domain  within  which  a  cost  curve  must  be 
detailed,  expressed  in  hours  per  student 
The  greatest  time  in  the  domain  of  detailing 
Incremental  resources  required  to  reach  from  P^^^ 
down  to  P^^^, 

Trainability  of  task  T  based  on  its  requirement 

for  cue/response  dimension  j  and  the  cue/response 

capability  of  a  simulator  option  m 

Training  time  for  task  T  on  device  i 

Time  scaling  factor  for  task  T 

number  of  tasks  over  which  the  simulator 

development  cost  should  be  amortized; 

Total  number  of  training  hours  on  the  actual 
equipment  (including  the  classroom)  in  order  to 
train  task  T  from  entry  to  standard  without  use  of 
a  simulator 


TWEAK  =  Acceleration  parameter  used  in  setting  SURCHG- 
TWT^  =  Economic  weight  for  task  T 

UBj^  =  Upper  bound  (constraint)  on  allocation  of  device  i 
usage  to  task  T 

Uj  =  Useful  time  ratio  for  task  T  on  the  actual 
equipment 

U^j  =  Useful  time  ratio  for  task  T  on  device  i 
UACT|  =  Calculated  use  per  year  of  device  i,  in  hours 
UBj^  =  an  upper  bound  to  device  i  allocations  for  each 
task  T 


UBENj^  =  Raw  (or  non-normalized)  benefit  score  of  simulator 
option  m  on  cue/response  dimension  j 
UBENjj^  =  Raw  benefit  score  of  most  capable  simulator  option 
on  cue/response  dimension  j 

UEST  =  Estimate  for  utilization  in  hours/year  for  device 
designed  by  Fidelity  Optimization  Module 
UMAXj  =  Maximum  number  of  hours  that  device  i  can  be  used 


in  a  year 

UNOMj  =  Nominal  utilization  in  hours  that  device  i  is  used 
in  a  year 

USMAX.  =  User-imposed  maximum  per-student  usage  of  device  i 
VCFAC  =  Factor  for  estimating  hourly  variable  cost  for  a 
training  device  design  from  the  investment  cost 
VHRC.  =  Variable  operating  cost  (per  hour)  of  device  i 
VHRC^  =  Variable  cost  (per  hour)  of  both  the  classroom  and 
the  actual  equipment  for  task  T 
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Table  6.  OSBATS  Variable  List 


^^*“AE(NO) 

VHRC,,(0)  = 

VHRC^g  = 
VHRC,„3  = 

VIOLTN  = 

VTOTC,  = 

woes  = 

WSR  = 
z  = 


Variable  cost  (per  hour)  of  the  actual  equipment 
in  the  non-operational  training  mode  (assumed  for 
now  to  be  six  times  VHRC(.„) 

Variable  cost  (per  hour)  of  the  actual  equipment 
in  the  operational  training  mode 
Variable  cost  (per  hour)  of  the  classroom 
Variable  cost  (per  hour)  of  the  full  mission 
simulator 

Flag  that  indicates  a  solution  that  violates 
constraints 

Total  variable  operating  cost  (per  hour)  of  device 
i,  including  investment  and  fixed  and  variable 
operating  costs 

Weight  for  the  importance  of  operations-cost 
savings  in  the  computation  of  ISIM^ 

Weight  for  the  importance  of  simulator 
requirements  in  the  computation  of  ISIM^ 
Intermediate  utility  variable  used  in  equations 
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System  Description 


The  description  of  a  single  node  in  the  model  consists  of 
three  components:  two  diagrams  (which  may  be  repeated)  and 
associated  explanatory  text.  The  diagram  for  the  node  being 
described  is  on  the  right-hand  side  of  the  page;  its  parent  is 
shown  on  the  left-hand  side.  The  text  is  written  beneath  the 
diagrams.  If  the  explanatory  description  requires  more  than  two 
pages,  both  parent  and  child  diagrams  are  repeated  on  the  next 
two  pages,  until  the  text  describing  each  node  is  completed.  The 
diagrams  describe  OSBATS  activities  from  the  viewpoint  of  the 
model  developer.  That  is,  they  show  the  data  required  by  model 
processes,  and  describe  in  detail  the  relationships  presumed  by 
the  model. 

OPT/A-Oi  Optimization  of  Simulation-Based  Training  Systems 
(OSBATS^ 

The  goal  of  the  OSBATS  Model  is  to  specify  the  designs  and 
concepts  of  use  for  training  devices  to  meet  training  require¬ 
ments  at  the  least  cost,  or  provide  the  greatest  training 
effectiveness,  given  the  cost.  Training  effectiveness  is 
measured  by  the  level  of  task  perfo2rmance  on  actual  equipment 
resulting  from  of  the  use  of  the  training  system.  Cost  comprises 
investment,  fixed  operating,  and  variable  operating  components. 

The  IDEFO  analysis  assumes  that  training  requirements  are 
defined  by  a  set  of  tasks  that  must  be  performed  to  prescribed 
standards  following  training.  Tasks  are  rated  along  several 
dimensions  prior  to  the  application  of  the  OSBATS  model. 

Training  devices  are  characterized  by  their  capability  to  present 
cues  and  collect  responses,  and  by  the  instructional  support 
features  they  possess.  The  population  of  instructional  features 
and  cue/response  dimensions  is  assumed  to  be  known  prior  to  the 
application  of  the  model,  and  is  represented  in  the  model  data 
base.  Training  devices  evaluated  by  the  OSBATS  model  may 
represent  currently  existing  devices,  or  they  may  be  generated  by 
the  design  components  of  the  OSBATS  model. 

The  model  considers  six  kinds  of  input  and  couLluI  data:  (1) 
Task  training  requirements,  (2)  Other  task  data  (3)  Training 
device  data,  (4)  Fidelity  dimension  data,  (5)  Instructional 
feature  data,  and  (6)  Training  system  data.  Data  from  each 
source  are  used  in  one  or  more  model  component. 

Task  training  requirements  describe  the  tasks  to  be  trained, 
specifying  entry  performance  level  and  training  standards,  safety 
and  special  performance  conditions,  and  cue  and  response  require¬ 
ments. 

The  two  components  of  other  task  data  describe  (l)  the 
training  time  that  would  be  required  to  meet  the  training 
requirements  in  one  hypothetical  case,  the  case  in  which  all 
training  is  conducted  in  a  classroom  or  on  actual  equipment,  and 
(2)  task  information-processing  characteristics. 
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Training  device  data  characterize  the  hypothetical  and 
actual  media  that  might  be  used  to  provide  training.  These  data 
describe  the  costs,  cue  and  response  capabilities,  and 
instructional  features  present  in  each  alternative  device. 

Fidelity  dimension  data  characterize  the  ways  that 
simulators  or  other  training  devices  may  differ  in  the  accuracy 
with  which  they  present  the  stimuli  and  response  options  from  the 
actual  equipment  and  operating  environment  in  which  the  tasks  are 
performed.  These  data  are  used  to  design  training  devices  that 
provide  the  optimal  levels  of  fidelity  on  each  fidelity  dimension 
as  a  function  of  training  device  cost. 

Instructional  feature  data  include  the  information  required 
to  select  the  instructional  features  that  provide  the  maximum 
improvement  in  training  efficiency  as  a  function  of  cost.  This 
segment  of  the  data  base  includes  rules  used  to  select 
instructional  features,  and  data  used  to  calculate  cost  and 
benefit  of  instructional  features. 

Finally,  training  system  data  include  general  information 
about  the  training  system,  and  other  miscellaneous  information 
required  by  the  model.  Specific  data  include  the  annual 
requirement  for  graduates  and  several  assumptions  used  by  the 
model . 
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/ 

OPT/AO:  Optimize  Training  System  Design 

The  first  level  of  decomposition  describes  the  three  problem 
areas  addressed  by  the  mociel :  (1)  training-concept  development, 

(2)  training  device  design,  and  (3)  assignment  of  training 
resources  to  devices.  Since  common  processes  are  involved  in 
several  of  these  problems,  the  analysis  includes  a  fourth 
component  that  conducts  the  preliminary  processing  required  for 
more  than  one  model  component.  The  next  level  of  decomposition 
enumerates  modeling  tools  addressing  the  problems. 

OPT/Al :  Perform  preliminary  processing.  This  activity 

produces  the  following  basic  information  that  is  processed 
further  by  several  model  components:  (1)  A  task-by- 
instructional-feature  matrix  that  specifies  which  instructional 
features  would  enhance  training  for  each  task,  (2)  a  matrix  that 
describes  the  cue  and  response  requirements  of  each  task  along 
each  fidelity  dimension,  (3)  task  weights  that  reflect  the 
relative  cost  of  training  each  task  on  actual  equipment,  and  (4) 
parameters  that  describe  the  course  of  learning  for  each  task  on 
each  candidate  training  device.  Although  this  activity  is 
preliminary  to  the  operation  of  any  specific  tool,  the  node  does 
receive  feedback  from  the  training-device  design  process  in  node 
0PT/A3.  That  is,  devices  designed  in  0PT/A3  will  need  to  be 
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Noie: 

ride:  Opcmize  Training  Systam  Design 

T5i«5r77577BB - 

1  Number  i 

OPT/AO 

1 OPT-2  •  1 

1  ; 

processed  by  this  node  to  provide  the  data  required  by  other 
modules . 

0PT/A2:  Develop  training  concept.  This  problem  area  is 

concerned  with  finding  clusters  of  tasks  that  have  similar 
training-device  needs.  These  tasks  form  training  requirements  to 
be  used  in  the  training-device  design  process  (OPT/ A3). 
Training-concept  development  occurs  early  in  the  training  sy3tem 
development  process,  and  is  refined  several  times  during  this 
process.  The  OSBATS  model  contains  a  single  tool  for  training- 
concept  development.  This  tool  evaluates  different  classes  of 
training  devices  (full-mission  simulators,  part-mission 
simulators,  actual  equipment)  that  may  meet  parts  of  the 
training  requirements. 

0PT/A3 ;  Design  training  devices.  This  problem  area  is 
concerned  with  designing  training  devices  that  provide  the 
optimal  training  effectiveness  given  the  investment  cost.  The 
principal  problems  addressed  in  the  model  are  (1)  providing  the 
optimal  level  of  fidelity  with  respect  to  presentation  of  the 
environment  and  equipment,  and  (2)  selecting  instructional 
features  that  are  tailored  to  the  tasks  being  trained  on  the 
training  device. 
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0PT/A4 ;  Assign  training  to  devices.  This  problem  is 
concerned  with  determining  which  training  device  or  devices 
should  be  used  to  train  each  task,  and  how  much  training  should 
be  conducted  on  each  device.  Two  tools  have  been  defined  for 
this  problem.  The  training-device  selection  tool  bases  the 
allocation  on  assumptions  that  provide  for  simpler,  interactive 
operation.  The  resource  allocation  tool  provides  a  more  detailed 
allocation  of  training  resources  to  training  devices,  that 
considers  constraints  on  training-device  use. 
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Task  Training  I  F  Fidelity  Rules 


Node: 
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OPT/Al ;  Perform  Preliminary  Processing 

This  activity  calculates  four  sets  of  variables  that  are 
used  by  other  model  components.  The  first  subactivity  applies  a 
set  of  rules  (5A)  that  examine  characteristics  of  the  tasks  to 
identify  effective  instructional  features.  The  second 
subactivity  applies  a  set  of  rules  {4D)  to  identify  task  cue  and 
response  requirements.  The  third  subactivity  determines  the 
total  cost  to  train  each  task  on  actual  equipment,  and  normalizes 
this  cost  to  be  used  as  a  task  weight.  The  fourth  activity  uses 
task  and  training-device  data  to  estimate  the  parameters  of 
training-device  learning  functions  for  each  task. 

OPT/All:  Analyze  tasks  for  instructional  features.  This 

activity  determines  the  instructional  features  that  are 
appropriate  for  each  task.  The  instructional  feature  rules  (5A) 
specify  the  kinds  of  tasks  for  which  each  instructional  feature 
is  appropriate.  The  conditions  of  these  rules  are  compared  to 
the  characteristics  of  the  tasks  ( 2B  and  lA)  to  specify  for  each 
task  the  set  of  applicable  instructional  features.  The  output  of 
this  activity  is  a  matrix,  [IFT.j.j^],  that  indicates  whether 
instructional  feature  k  will  enhance  the  training  efficiency  of 
task  T.  An  element  IFT^j^  of  this  matrix  contains  the  value  1.0 
if  instructional  feature  k  is  appropriate  for  task  T.  This 
activity  is  conducted  as  follows. 


78 


I  TTM*  PfQOMSmQ 


Omsm\  7/27/m  1  NurMt 


First,  a  rule  base  is  interrogated  to  compare  task  data  and 
rule  conditions.  The  instructional  feature  rules  (5A)  associate 
instructional  features  with  specific  task  characteristics.  An 
example  of  such  a  rule  is  given  below: 

IF:  Entry  performance  ( lAl )  <  0.4, 

and  Intrinsic  feedback  (2B11)  is  absent, 
and  The  task  involves  continuous  movement  (2B1), 
or  procedures  (2B2), 

or  decision  making/rule  using  (2B4), 

THEN:  Automated  Performance  Alerts  is  indicated  for  this  task. 

This  activity  compares  the  conditions  of  the  Instructional 
feature  rules  to  the  task  characteristics  (2B)  and  task  learning 
points  (lA),  and  identifies  matches,  which  it  passes  on  to  the 
n^xt  activity. 

Next,  this  activity  takes  the  matches  produced  in  the 
interrogation,  and  sets  the  corresponding  cells  of  IFT^j^.  The 
IFT  matrix  is  defined  as  follows: 

Tpm  s  J  ^  ®  match  was  found  between  feature  k  and  task  T 

^'”Tk  *  I  0  otherwise. 
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0PT/A12:  Analyze  tasks  for  fidelity  features.  This 

activity  derives  the  task  cue  and  response  requirements  (FRQTyj) 
on  each  fidelity  dimension.  The  derivation  is  controlled  by  the 
information  in  the  fidelity  rule  base  (4D).  The  fidelity  rule 
base  calculates  the  FRQT^^  based  upon  the  values  of  variables 
that  describe  task  activities  (2C)  and  information  processing 
requirements  (2B).  The  task  activity  variables  are  situation  and 
domain  dependent.  The  rule  base  is  hierarchically  organized  and 
operates  using  backward  chaining. 

The  following  example  shows  one  of  the  rules: 

IF:  Performance  cues  are  provided  by  longitudinal 

acceleration, 

and  Motion  cues  should  be  provided  by  platform  motion  (as 
opposed  to  seat  motion) , 

and  The  magnitude  of  the  longitudinal  acceleration  cues  is 
moderate  or  high, 

and  The  task  requires  the  performance  of  an  emergency 
procedure , 

and  The  platform  longitudinal  acceleration  provides  a  cue  for 
the  initiation  of  the  emergency  procedure 

THEN:  Platform  surge  is  required  for  training  (the  requirement 

for  platform  motion  is  given  the  value  0.9). 
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‘  OPT-3 

Some  of  the  conditions  for  this  rule  are  derived  from  the  value 
of  other  data;  other  conditions  are  direct  data  values.  The 
resulting  values  of  FRQT.j.j  are  in  the  interval  [0,  1]. 

0PT/A13:  Determine  task  weights.  This  activity  calculates 

a  set  of  weights  that  are  used  to  compare  the  importance  of 
training  Improvements  for  different  tasks.  The  weights  are  based 
on  the  cost  of  training  the  tasks  on  actual  equipment .  The 
processes  contained  in  this  activity  calculate  the  investment 
(0PT/A131),  fixed  operating  (0PT/A132),  and  variable  operating 
(0PT/A133)  components  of  the  cost  of  training  on  actual 
equipment.  The  final  process  (0PT/A134)  combines  and  normalizes 
the  cost  estimates  to  produce  task  weights. 

0PT/A14:  Determine  learning  function.  The  basis  for  the 

determination  of  training  effectiveness  is  a  learning  function 
that  relates  performance,  Pxi(t),  to  task  and  device  variables. 

We  use  a  power  function  of  the  following  form  to  describe 
learning : 

P,j,j(t)  =  ASM^i  {1  -  [1  +  TMTiTSFT{HSTi  +  t)]"^}. 
where 
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t  =  the  training  time , 

T  =  the  task  to  be  trained, 
i  =  the  training  device, 

=  the  asymptote  of  the  learning  curve, 

TMt^  =  the  time  multiplier  for  the  particular  device  and  task, 
TSFj  =  the  time  scaling  factor  for  the  task, 

HSii  =  the  head  start  for  the  device  and  task,  and 
r  =  the  learning  curve  exponent. 

The  power  function  form  is  used  because  of  the  good  fit  such  a 
function  has  provided  to  empirical  data  (Newell  &  Rosenbloom, 
1981).  For  actual  equipment  the  parameters  of  this  function  may 
be  based  on  fits  to  empirical  research  or  on  fits  to  learning 
curves  estimated  by  subject  matter  experts  ( SMEs ) .  For  notional 
equipment  a  procedure  has  been  developed  (0PT/A14)  for  estimating 
the  learning  curve  parameters  by  extrapolation.  The  processes  in 
this  activity  first  calculate  some  of  the  basic  parameters  needed 
to  calculate  the  asymptote  of  the  learning  function  (0PT/A141). 
Then  specific  processes  estimate  the  values  for  the  parameters 
that  do  not  depend  on  the  training  device  (OPT/A142),  and  those 
that  do  depend  on  the  training  device  (0PT/A143),  respectively. 
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j  Data:  7/27/88  Number 
:  OPT-3 


0PT/A13:  Determine  Task  Weights 

The  task  weights  are  used  as  a  measure  of  Importance  of  the 
tasks  by  several  of  the  modules.  They  represent  the  normalized 
life  cycle  cost  of  training  the  tasks  to  standard  through  the  use 
of  classroom  activities  and  actual  equipment  exercises  only.  The 
elements  of  these  weights  come  from  the  actual  equipment  training 
times  and  other  costs  (2A),  the  actual  equipment  cost  data  {3A), 
and  training  course  and  system  information  (6A).  The  weights  are 
calculated  in  four  steps. 

0PT/A131;  Compute  the  investment  allocation.  The 
investment  in  the  actual  equipment  is  determined  by  finding  the 
number  of  actual  equipment  devices  needed  to  produce  the  yearly 
graduates  (GRADS),  and  then  allocating  this  investment  over  the 
equipment  life  cycle  and  the  tasks  using  the  device  in  proportion 
to  their  usage  (classroom  investment  is  ignored).  The  total 
training  hours  for  each  task  on  the  actual  equipment  is 

TAEit.  =  NOH-r  +  0H.J.  +  SUHy 

where  the  parameters  on  the  right  hand  side  are  data  or  SME 
estimates  available  in  the  data  base: 
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NOH^j  =  Number  of  non-operational  hours  on  actual  equipment  to 
train  task  T  from  entry  to  standard  without  a  simulator 
(2A2) 

OHj  =  Number  of  operational  hours  on  the  actual  equipment  to 
train  task  T  from  entry  to  standard  without  a  simulator 
{2A3) 

SUH^  =  Setup  hours  on  the  actual  equipment  for  task  T  { 2A4 ) . 

Since  the  maximum  annual  utilization  of  actual  equipment  is 
UMAX^g  (3A5),  the  number  of  actual  equipment  devices  per  year 
(Nae)  is: 

Nae  =  ceiling((  Z  TAE.p)  *  GRADS / UMAX ae )  ' 


v/here 

ceiling (argument)  =  the  smallest  integer  at  least  as  great  as  the 

function  argument. 

The  proportion  of  actual  equipment  hours  per  task  (TAENRM-p)  is: 

TAENRM^  =  TAEt/(  Z  TAE^ ) 

T 
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The  allocated  annual  investment  cost  to  each  task  (CINV-p)  is: 

CINVij  =  <  INV;^g/LC^g)  *  N;^g  •  TAENRMj. 

OPT/A132:  Compute  the  fixed  operating  cost  allocation .  The 

allocation  of  fixed  operating  cost  of  the  actual  equipment  (CFX.J.) 
is  determined  by  proportioning  the  fixed  operating  cost  (FOC^^E' 
3A2)  over  the  tasks  using  the  device  in  accordance  to  their 
usage.  (Classroom  fixed  operating  cost  is  assumed  negligible 
when  apportioned  on  a  per-student  basis.)  That  is, 

CFX.J,  =  FOC^g  *  N^g  *  TAENRMiji. 

OPT/A133:  Compute  the  variable  operating  cost  allocation. 

The  yearly  variable  operating  cost,  CVAR^,  allocated  to  training 
task  T  is  just  the  total  variable  operating  cost  for  each  task, 
including  other  task  training  cocts  (such  as  ammunition) : 

eVARj  =  [(VHRC.j.^^g  *  TTOTj)  +  OTC-j.]  *  GRADS 

where  TTOTm  is  the  total  numher  rf  training  hours  on  the  actual 
equipment  (including  the  classroom)  In  order  to  train  task  T  from 
entry  to  standard  without  use  of  a  simulator,  VHRCj  ^g  is  the 
variable  cost  (per  hour)  of  both  the  classroom  and  the  actual 
equipment  for  task  T,  and  OTCij  represents  other  costs  of  training 
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task  T,  such  as  ammunition.  OTCj  is  available  from  the  data 
base  (2A5).  TTOT-j  are  computed  as  follow: 

TTOTy  =  CRH^  +  TAE^ 

VHRCt^AE  *  C(CRHt  •  VHRCcr)  +  (NOHt  *  VHRCaE(No)^ 

+  ((OHj.  +  SUHj)  *  VHRC;^g(Q)  )  ]/TTOTt 

where  the  following  are  found  in  the  data  base  or  estimated  using 
ad  hoc  rules: 

CRH(j  =  Number  of  classroom  hours  needed  to  train  task  T 
in  conjunction  with  the  actual  equipment  from 
entry  to  standard  without  a  simulator  ( 2A1 ) 
VHRC^£(rq)  =  Variable  cost  (per  hour)  of  the  actual  equipment 
in  the  non-operational  training  mode  (assumed  for 
now  to  be  six  times  VHRCq^) 

=  Variable  cost  (per  hour)  of  the  actual  equipment 
in  the  operational  training  mode  (3A3) 

VHRCqj,  =  Variable  cost  (per  hour)  or  the  classi’ocm  ( 3A3 )  . 

OPT/A134:  Compute  and  normalize  the  economic  weights.  The 

investment,  fixed  operating  and  variable  operating  allocations 
are  summed  to  yield  the  total  life  cycle  training  cost  for  each 
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task  {TCST.J.),  which  is  then  normalized  to  sum  to  one  across  all 
tasks  to  produce  the  task  weights  (TWT-p). 

TCST^  =  CINV^  +  CFX^  +  CVAR^ 

TWTj.  =  TCSTj/{  I  TCSTj) 
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OPT/A14;  Determine  Learning  Function 

The  parameters  of  the  learning  function  depend  upon  the 
entry  performance  level  and  training  standard  (lA),  the  estimated 
training  time  on  actual  equipment  (2A),  task  cue  and  response 
requirements  (OIB) ,  training  device  cue  and  response  capabilities 
(3B)  and  instructional  features  (3C),  the  instructional  feature 
appropriateness  matrix  (OlA),  and  several  assumptions  (6B).  The 
learning  curve  parameters  are  estimated  in  three  steps.  In  the 
first  step,  several  intermediate  values  that  are  used  to 
calculate  transfer  of  training  are  determined.  In  the  second 
step,  the  time  scaling  factor  and  actual  equipment  head  start  are 
determined.  The  results  of  the  first  two  steps  are  used  in  the 
third  step  to  determine  device  learning  curves. 


0PT/A141:  Determine  transfer  ratio  parameters .  The 

asymptotes  of  the  device  learning  curves,  ASM^^ ,  will  be 
calculated  in  a  later  activity  (OPT/A1431)  according  to  the 
following  equation: 


ASMti  =  q^ 


(FCAP 


IJ 
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where  the  Greek  letter  "pi"  indicates  a  product  (multiplication) 
of  all  the  terms  that  follow,  and  where 

FCAPij  =  the  cue  and  response  capability  of  device  1  on  fidelity 
dimension  j, 

FRQTipj  »  the  cue  and  response  requirement  of  task  T  on  fidelity 
dimension  j,  and 

q-j>,  bjj ,  and  =  scaling  constants  to  adjust  the  range  of  the 
result . 

The  scaling  constants  adjust  the  result  to  be  in  the  range  [0,1], 
such  that:  (1)  Perfect  fidelity  in  all  dimensions  leads  to  an 
asymptote  of  1.0;  (2)  Meeting  all  requirements  of  a  task  exactly 
produces  an  asymptote  equal  to  the  training  standard  adjusted  by 
a  certain  percentage  (this  benchmark  training  standard  is 
calculated  in  0PT/A1411);  (3)  A  device  with  perfect  fidelity  in 
all  dimensions  except  one,  and  no  capability  in  the  remaining 
dimension  will  produce  an  asymptote  equal  to  the  minimum 
performance  parameter  for  that  dimension  (4C). 

This  activity  calculates  the  scaling  constants  that  will  be 
used  in  the  equation  to  calculate  the  learning  curve  asymptote. 
The  specific  procedures  used  in  this  activity  estimate  the 
benchmark  transfer  (0PT/A1411),  equation  exponent  (0PT/A1412), 
reference  point  (0PT/A1413),  and  equation  multiplier  (0PT/A1414). 
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A  mathematical  derivation  of  the  equations  that  are  used  is 
provided  in  Appendix  B. 

OPT/A142:  Determine  common  learning  parameters.  This 

activity  calculates  the  value  of  the  time  scaling  factor,  TSP™ , 
(in  0PT/A1421)  and  the  head  start  for  actual  equipment,  HS71  » p , 
(in  OPT/A1422)  to  give  an  empirical  fit  to  available  training 
program  data.  The  specific  procedure  is  described  in  the 
detailed  diagram  for  this  activity. 

0PT/A143:  Determine  specific  learning  curve  parameters . 

This  activity  calculates  the  device-specific  parameters  of  the 
learning  curve.  First,  it  applies  the  equation  described  in 
0PT/A14  to  calculate  the  asymptote  of  the  device  curves 
(0PT/A1431).  Then  It  calculates  the  time  multiplier  (TM^^)  and 
head  start  for  each  training  device  (0PT/A1432  and 

0PT/A1433,  respectively).  The  specific  procedure  is  described  in 
the  detailed  diagram  for  this  activity. 
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UPT/A141 : 

Determine  Transfer 

Ratio  Parameters 

The  parameters  of  the  equation  used  to  determine  the 
asymptote  of  the  d-^vice  learning  curves  are  determined  in  four 
steps.  First  a  benchmark  transfer  value  (ASM^j^)  is  determined  by 
adjusting  the  training  standard.  Then  the  exponent  of  the 
equation  (Q-j-)  is  determined  numerically  to  ensure  that  the 
asymptote  for  a  device  that  exactly  meets  all  cue  and  response 
requirements  equals  Then  the  fidelity  dimension  reference 

points  (b-pj)  are  determined  to  set  the  maximum  effect  each 
fidelity  dimension  may  have  on  the  ASM^^ .  Finally,  the  equation 
multiplier  (q.p)  is  determined  to  normalize  the  equation  to  have 
the  maximum  value  of  l.O. 

The  equation  for  ASMipj^  has  several  properties  that  make  it 
desirable.  (1)  A  training  device  that  meets  all  cue  and  response 
requirements  will  produce  an  asymptote  that  allows  it  to  be  used 
to  train  to  the  training  standard  in  finite  time,  (2)  Fidelity 
dimensions  with  high  requirements  are  more  important  in  the 
determination  of  the  asymptote  than  dimensions  with  low 
requirements.  (3)  The  overall  importance  of  a  dimension  may  be 
limited  by  the  values  of  b.j.j  . 
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0PT/A1411:  Determine  benchmark  transfer.  The  benchmark 

transfer  (ASM.j<p)  is  the  asymptote  that  is  associated  with  the 
hypothetical  training  device  that  exactly  meets  all  cue  and 
response  requirements.  It  is  obtained  by  adjusting  the  training 
standard  by  a  constant  proportion,  as  follows: 

ASMtr  =  STD.J.  +  SADJ  (1  -  STDj) 

where 


STD.J  =  the  training  standard  for  task  T,  and 
SADJ  *  the  adjustment  proportion. 


0PT/A1412:  Determine  equation  exponent.  The  exponent  of 

the  equation  (Q^)  is  set  to  ensure  that  the  asymptote  for  a 
device  that  exactly  meets  all  cue  and  response  requirements  is 
ASMtp.  This  condition  may  be  expressed  algebraically  as  follows: 


TR 

where 


FRQT^j  +  ( 1 


1 

FRQT^j)  FMINj  FRQT^^ 


FRQT.j.^ 
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FMINj  =  the  minimum  performance  of  a  device  with  cue  and 

response  capability  of  0.0  on  dimension  j,  i.O  on  all 
other  dimensions,  and 

FROTijij  =  the  cue  and  response  requirement  for  task  T  on 
dimension  j. 

The  solution  may  be  found  numerically  using  bisection.  However, 
when 

ASMtr  <  1  I  FMIN.  ^ 

then  the  FMINj  and  the  FRQT.j.j  are  inconsistent,  and  no  value  of 
Qj  exists.  In  this  case,  the  FMINj  must  be  lowered  for  tasks 
with  Inconsistencies  until  a  solution  is  found.  To  eliminate  the 
inconsistencies,  the  FMINj  are  multiplied  by  the  following 
factor : 


*FMIN  . 


FMIN^^  FRQTy^) 

1/JC^  ■ 

- 

ASMtj,  J 

l/{ 1-FRQTjj) 


where 
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JCj  =  the  number  of  fidelity  dimensions,  j,  such  that  FRQTm^ 

>  0  and  FMINj  >0. 

0PT/A1413:  Determine  reference  point.  The  cue  and  response 

reference  point  (b^j)  is  a  variable  that  is  included  in  the 
asymptote  calculation  so  that  the  asymptote  is  FMINj  in  the 
appropriate  situations.  The  b-jj  are  calculated  using  the 
following  formula: 


0PT/A1414:  Determine  equation  multiplier.  The  multiplier 

of  the  equation  that  determines  the  asymptote,  q™,  normalizes  the 
equation  to  have  a  maximum  value  of  1.0.  The  multiplier  is 
calculated  using  the  following  equation: 
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OPT/A142:  Determine  Common  Learning  Curve  Parameters 


This  activity  calculates  the  time  scaling  factor  (TSF.p)  for 
each  task  and  the  head  start  for  actual  equipment. 

OPT/A1421:  Determine  time  scaling  factor.  By  convention, 

the  time  multiplier  parameter,  TMj^  for  actual  equipment  is  1. 
Training  data  for  actual  equipment  are  used  to  calculate  the 
time  scaling  factor,  TSF,j.,  using  the  following  formula.  We 
assume  that  the  learning  curve  exponent,  r,  is  0.7  to  be 
consistent  with  the  human  factors  literature  (e.g.,  Card,  Moran, 
&  Newell,  1983)  and  with  theoretical  considerations  (Newell  & 
Rosenbloom,  1981). 


TSF^  = 


(1  -  STD,j.)"^''^  -  (1  -  ENT^) 

ttotI 


where 


ENTip  =  the  entry  level  performance  on  task  T 
STDp  ®  the  training  standard  for  task  T,  and 
TTOTp  *  the  total  number  of  training  hours  on  the  actual 
equipment  to  train  to  the  standard  on  task  T. 
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The  time  scaling  factor  Is  related  to  the  movement  in  performance 
between  entry  and  the  training  standard  that  is  associated  with 
an  hour  of  training  time  on  the  actual  equipment. 

0PT/A1422:  Determine  actual  equipment  head  start .  The  head 

start  for  actual  equipment  may  be  calculated  in  this 

node  because  both  TMj^  and  ASM.t.^  are  equal  to  1.0  for  the  actual 
equipment.  Under  these  conditions,  the  following  formula  is 
equivalent  to  the  formula  presented  in  0PT/A1433. 


(1  -  ENT^)  -  1 


HS  —  -  * 

T,AE  “  TSF.J 


With  this  head  start  included  in  the  learning  curve  for  actual 
equipment,  a  student  who  trains  for  zero  time  will  still  score 
performance  ENT-j.  on  task  T. 


0PT/A143:  Determine  Specific  Learning  Curve  Parameters 


This  activity  calculates  the  learning  curve  parameters  that 
are  specific  to  both  the  task  and  the  training  device.  These 
parameters  include  the  asymptote  the  time  multiplier 

(TMTi),  and  the  head  start  (HS.pj^).  Each  of  the  following  three 
subactivities  determines  one  of  these  parameters. 


0PT/A1431;  Determine  transfer  ratio.  The  asymptote  or 
transfer  ratio  (ASM^^)is  calculated  from  the  transfer  parameters 
determined  in  0PT/A141,  the  task  cue  and  response  requirements 
(FRQTmj),  and  the  device  cue  and  response  capabilities  (FCAPj^j), 
according  to  the  following  formula: 


ASM^^  =  q.j,  I  J  (FCAP^j  +  b^^ ) 
where 

FCAPj^j  =  the  cue  and  response  capability  of  device  i  on  fidelity 
dimension  j, 

FRQT.J.J  =  the  cue  and  response  requirement  of  task  T  on  fidelity 
dimension  J, 
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qiji  =  the  equation  multiplier  (0141A4), 
bijj  *  the  fidelity  dimension  reference  point  (0141A3),  and 
=  the  equation  exponent  (0141A2). 


0PT/A1432:  Determine  time  multiplier.  The  time  multiplier 

reflects  the  improvement  in  training  efficiency  that  may  occur 
through  the  use  of  a  training  device.  This  improvement  may  occur 
because  setup  time  is  reduced,  or  may  result  from  the  proper  use 
of  instructional  features.  Several  assumptions  are  used  to 
predict  the  extent  of  efficiency  improvement.  The  processes  in 
this  activity  combine  the  effect  of  instructional  features 
(determined  in  0PT/A14321)  and  reduced  setup  time  (determined  in 
0PT/A14322  and  0PT/A14323)  to  determine  the  time  multiplier  (in 
0PT/A14324).  The  detailed  diagram  for  this  activity  describes 
both  the  procedure  and  assumptions  used  in  this  activity. 


0PT/A1433:  Determine  head  start.  The  head  start  (HS.p^)  is 

set  to  ensure  that  the  entry  performance  level  is  obtained  for 
the  time,  t  =  0.  HSipj^  is  calculated  using  the  following  formula; 


HS 


Ti 


1 

TM.j.^  TSF^ 


(1  -  ENT^/ASM^^)  -  1 


where  ENT.j  <  ASM^^,  and 
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the  time  multiplier  for  task  T  and  device  i  (01D3B), 
the  time  scaling  factor  for  task  T  (OlDl), 
the  entry  performance  level  for  task  T  ( lAl ) , 
the  asymptote  for  task  T  and  device  i  (01D3A),  and 
the  exponent  of  the  learning  function  (assumed  to  be 
0.7)  . 


is  undefined  when  ENTj  >  ASM.jj^,  because  this  condition 
indicates  that  is  impossible  to  obtain  operational  performance 
even  as  great  as  the  entry  performance  level  using  training 
device  i. 
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OPT/A1432:  Determine  Time  Multiplier 

The  time  multiplier  (TMijjj^)  measures  the  increased  efficiency 
of  device-based  training  due  to  instructional  features  and 
reduced  setup  time.  This  activity  is  not  carried  out  for  actual 
equipment,  since  the  time  multiplier  for  actual  equipment  is  1.0. 

0PT/A14321:  Determine  instructional  feature  improvement . 

The  estimated  instructional  feature  improvement  in  efficiency 
(lADJijj^)  is  affected  by  two  limits.  The  first  limit  reflects  the 
maximum  number  of  appropriate  instructional  features  that  would 
be  of  benefit  to  training  (MAXIF;  6B4).  A  device  with  more  than 
MAXIF  instructional  features  would  not  get  -my  additional  benefit 
over  a  device  with  MAXIF  features.  The  second  limit  is  the 
maximum  percer'+'age  improvement  in  training  efficiency  due  to 
instructional  features  (IFBN;  6B3).  A  device  with  MAXIF 
appropriate  instructional  features  would  produce  a  proportional 
decrease  in  training  time  of  IFBN.  The  first  formula  in  the 
calculation  of  lADJ^^  calculates  the  number  of  appropriate 
instructional  features  from  the  instructional  feature 
appropriateness  matrix,  IFTq.j^,  (OlA)  and  the  list  of 
instructional  features  possessed  by  each  device,  DVIF^j^  (3C). 
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IFNM^^  =  min  [[  ^  ] 

k 

The  final  adjustment  value  (lADJ^j)  is  then  determined  using  the 
following  formula. 

_ 1 _ 

lADJ^i  =  1  -  j  (IFNM.j,^/MAXIF)^^^  IFBN  ] 

This  equation  produces  an  adjustment  such  that  each  additional 
instructional  feature  has  a  smaller  impact  on  lADJ^^  than  the 
previous  instructional  feature. 

0PT/A14322:  Determine  actual  equipment  useful  time  ratio . 

The  actual  equipment  useful  time  ratio  {Uy  represents  the 
percentage  of  use  of  the  equipment  in  whicfi  effective  training 
occurs.  That  is, 


U 


T,AE 


CRH^  +  NOH^  +  OH^ 
TTOT^ 


The  numerator  in  this  expression  represents  the  time  the  student 
is  actually  undergoing  instruction  in  the  classroom  (CRH-p),  on 
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the  actual  equipment  prior  to  operation  (NOHj),  and  on  the  actual 
equipment  during  operation  (OHj).  The  denominator  represents 
this  total  Instruction  time  plus  setup  hours,  SUH^.  Setup  hours 
include  such  things  as  student  time  in  being  transported  from 
classroom  to  airfield,  checking  out  flight  equipment,  waiting  for 
takeoff  clearance,  or  performing  tasks  that  are  already  known  to 
the  performance  standard,  l.e.,  obligatory  student  time  during 
which  no  instruction  is  being  provided. 

UPT/A14323:  Determine  device  useful  time  ratio.  The  device 

useful  time  ratio  represents  the  percentage  of  use  of  the 

device  in  which  effective  training  occurs.  The  formula  used 
assumes  that  some  percentage  (SUSAV;  6B5 )  of  the  setup  time  is 
saved  when  training  is  conducted  in  a  training  device.  That  is. 


CRH.J,  +  NOH,j,  +  OH.J, 

TTOT^  -  SUH^  *  SUSAV 


0PT/A14324:  Calculate  time  multiplier.  The  time  multiplier 

(TMti)  is  calculated  from  the  previous  results  using  the 
following  formula: 
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It  represents  an  improvement  in  the  usage  of  time,  compared  to 
the  usage  of  time  on  actual  equipment,  due  to  improved 
instructional  features  (lADJ-pj^)  or  due  to  a  reduction  in  "wasted" 
setup  time 
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0PT/A2 :  Develop  Training  Concept 

This  component  Is  concerned  with  methods  to  cluster  tasks 
that  have  similar  training-device  needs.  The  output  of  this 
component  is  a  set  of  preliminary  training-device  requirements 
that  can  be  used  as  the  basis  of  che  training-device  design 
process  in  0PT/A3 .  We  have  currently  developed  a  single  tool  for 
this  activity,  a  tool  that  evaluates  general  training  system 
alternatives  including  full-mission  simulators,  part-mission 
simulators,  and  actual  equipment.  Because  there  is  only  one  tool 
in  this  component,  this  level  of  decomposition  in  the  IDEFO  model 
is  not  required.  However,  ic  was  included  for  two  reasons. 

First,  adding  this  level  allows  us  to  represent  all  tools  at  the 
same  level  of  decomposition  in  the  overall  model.  It  is  hoped 
that  this  parallel  structure  will  make  the  numbering  of  model 
processes  easier  to  understand.  Second,  including  this  level  of 
decomposition  provides  a  place  holder  for  future  tools  that  might 
be  developed  for  this  training-concept  development. 

0PT/A21:  Recommend  simulator  configuration.  This  tool 

evaluates  general  training  system  alternatives.  It  considers  the 
following  three  classes  of  training  device:  (1)  full-mission 
simulators,  (2)  part-mission  simulators,  and  (3)  actual 
equipment.  The  tool  evaluates  these  alternatives  to  provide 
guidance  regarding  the  types  of  device  that  would  be  appropriate 
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^nr  the  training  requirements.  The  recommendations  are  based  on 
i-  B  task  requirements  and  estimates  of  the  cost  to  meet  these 
requirements.  The  activity  contains  two  subactivities,  evaluate 
each  task  (0PT/A211)  and  develop  simulation  recommendations 
(OPT/A212) . 
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OPT/A21:  Recommend  Simulator  Configuration 

This  tool  examines  the  need  and  cost-effectiveness  of  using 
either  a  full-mission  simulator  (PMS)  or  one  or  more  part-mission 
simulators  (PMSs)  to  replace  training  time  in  the  actual 
equipment.  Built  into  the  evaluation  performed  by  this  tool  is 
the  examination  of  simulator-unique  capabilities,  such  as 
training  in  unsafe  situations,  and  cost  savings  to  establish  the 
value  of  training  with  some  sort  of  training  device.  In 
addition,  the  development  cost  of  the  training  device  that  would 
be  required  to  achieve  these  benefits  is  examined.  Using  these 
results  the  module  partitions  the  set  of  training  requirements 
into  the  following  three  subsets:  (1)  a  subset  for  which  an  FMS 
should  be  designed  by  the  later  modules,  (2)  a  subset  requiring 
one  or  more  PMSs  to  meet  the  training  needs,  (3)  a  subset 
requiring  training  on  actual  equipment. 

0PT/A211:  Evaluate  each  task.  This  first  process  is  a  set 

of  operations  that  acts  upon  each  task;  it  is  repeated  for  every 
task.  The  requirements  for  training  by  simulator  and  the 
operating  cost  savings  associated  with  simulator  training 
(0211A)  are  first  examined  for  each  task  (in  0PT/A2111).  Then 
the  task  cue  and  response  requirements  to  achieve  these  results 
are  used  to  provide  a  preliminary  estimate  of  training-device 
development  costs  (021 IB;  evaluated  in  OPT/A2112). 
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OPT/A212:  Develop  simulation  recommendations.  This  second 

process  then  operates  on  the  simulation  requirement  index  (0211A) 
and  development  cost  index  (021 IB)  from  the  previous  process  to 
generate  a  recommendation  concerning  the  need  for  a  full-mission 
simulator  versus  several  possible  part-mission  simulators.  This 
recommendation  compares  the  development  cost  index  with  the 
potential  operating  cost  savings  over  the  equipment  lifecycle  to 
recommend  simulation-based  or  actual-equipment-based  training. 

The  development  cost  index  is  used  to  recommend  training  on  an 
FMS  or  PMS  for  those  tasks  for  which  simulation-based  training  is 
indicated.  The  first  subprocess  of  this  activity  (OPT/A2121) 
creates  a  scatterplot  that  shows  the  simulation  requirement  index 
and  development  cost  index  for  each  task.  The  second  subprocess 
(0PT/A2122)  summarizes  the  recommendations. 
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OPT/A211:  Evaluate  Each  Task 

Evaluate  simulation  indicators  and  evaluate  fidelity 
requirements  are  the  two  processes  needed  for  this  activity. 

These  processes  generate  the  data  that  are  needed  to  develop 
simulation  recommendations  (OPT/A212). 

0PT/A2111:  Evaluate  simulation  indicators.  The  two  classes 

of  simulation  indicators,  requirements  and  operating-cost 
savings,  determine  the  extent  to  which  a  simulator  of  any  kind 
would  be  benerlcial  for  training  on  a  task-by-task  basis.  These 
indices  are  developed  by  aggregating  SME  judgments  on  a  variety 
of  factors  associated  with  simulator  value.  The  indicators  are 
used  to  justify  expenses  in  the  development  of  simulation 
capabilities.  The  simulation  requirements  are  determined  in 
0PT/A21111;  the  operating-cost  savings  is  determined  in 
OPT/A21112. 

OPT/A2112:  Estimate  Development  Costs.  The  fidelity 

requirements  of  each  task  quantify  the  relative  need  for  a  full- 
mission  simulator  compared  to  part-mission  simulators.  In 
addition,  tasks  that  require  a  simulator  to  represent  a  variety 
of  different  cues  and  responses  to  a  high  level  of  fidelity  will 
require  a  more  expensive  simulator  to  train  effectively.  Thus, 
the  fidelity  dimension  data  can  be  used  to  compute  the 
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development  cost  required  to  train  the  task  on  a  simulator.  The 
activity  proceeds  by  first  estimating  the  cost  to  meet  the 
requirements  for  each  fidelity  dimension  (OPT/A21121)  and  then 
summing  the  costs  across  fidelity  dimensions  (OPT/A21122 ) . 
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0PT/A2111;  Evaluate  Simulation  Indicators 

This  analysis  results  in  the  quantification  of  a  simulation 
indicator  (0211A)  for  each  task.  This  simulation  indicator 
reflects  both  the  requirement  for  simulation  to  train  the  task 
(021111A)  and  the  cost  savings  that  would  be  expected  if 
simulation  were  employed  (021112A). 

0PT/A21111:  Rate  simulation  requirements.  This  analysis  is 

completed  by  using  the  data  for  simulation  requirements  (IB). 
These  data  were  developed  using  the  following  11  conditions: 

(a)  training  without  simulation  is  impossible 

-  task  involves  imposing  an  emergency  (e.g.,  malfunction) 
that  places  the  student  and  equipment  in  danger 

-  task  involves  a  procedure  that  violates  safety 
standards  for  the  actual  equipment 

-  task  involves  a  violation  of  current  regulations  or 
restrictions  on  operations 

-  task  involves  response  to  specific  battle  situations 
that  cannot  be  provided  in  the  real  world 

“  trainees  inexperience  makes  task  unsafe 

-  task  involves  response  to  a  component  failure  that 
cannot  be  induced  without  undue  hazard 


114 


Simulation 

Requirementa 


IB 

7 


Task  Training 
Hours  and  Coats 
Learning 

Curves  _ 


2A 


Learning 
Points 
I  1A 


01D 


Baseline  Vatiabla 
Operating  Coats 
Training  Device  Costs 


01E 


3A 


Operating 
Coat  Savinga 


OperaPng 
C^  Savings 


021112B 


Simulation 

Requirements 


021111A 


Operating 
Cost  Savinga 
021112A 


Training  System  Data 
or  User  Judgement  Reqmta/ 
Gps  Cost  Weignts 


Node: 

1  TWa:  Evaluate  Simulation  indicators 

Data:  7/12/88 

Numter  1 

0PT/A2111 

1 

OPT-13 

(b)  training  without  simulation  is  difficult  or  costly 

-  task  involves  operation  in  special  weather  or  lighting 
conditions  that  do  not  occur  naturally  often  enough 

-  task  involves  response  to  specific  battle  situations  or 
other  special  situations  that  are  very  expensive  in  the 
real  world 

-  task  Involves  training  in  equipment  that  is  not 
accessible  enough 

(c)  training  would  be  significantly  improved  with  simulation 

-  simulation  offers  special  training  effects  that  enhance 
acquisition  or  transfer 

-  number  of  trials  per  unit  time  can  be  greatly  increased 
with  simulation. 

If  any  condition  in  group  (a)  pertains  for  a  given  task, 
simulation  is  considered  an  absolute  requirement  and  a  rating  of 
1.0  is  assigned  to  the  index  for  simulation  requirements,  ISIMip. 
If  no  condition  from  group  (a)  pertains,  the  number  of  conditions 
in  groups  (b)  and  (c)  determine  the  rating.  One  condition  in 
group  (b)  yields  a  rating  of  0.4,  two  yield  a  0.6,  and  three 
yield  a  0.7.  One  condition  in  group  (c)  is  worth  0.2,  and  two 
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are  worth  0.3.  The  ratings  from  groups  (b)  and  (c)  are 
additive.  If  no  conditions  pertain  for  a  given  task,  then  a 
ratings  of  0.0  is  assigned. 


OPT/A21112:  Calculate  operating-cost  savings.  This 

analysis  yields  an  index  for  operating-cost  savings,  the  second 
valid  reason  to  use  a  simulator.  These  operations-cost  savings 
are  associated  with  replacing  some  training  hours  in  the  actual 
equipment  with  training  hours  in  a  simulator.  The  three 
operations  in  this  process  minimize  operating  cost  using  a 
combination  of  training  on  actual  equipment  and  training  on  a 
defined  simulator  (OPT/A211121 ) ,  calculate  the  operating-cost 
savings  of  the  minimal  cost  option  over  training  on  actual 
equipment  alone  (OPT/A21H22 ) ,  and  convert  this  savings  to  a 
savings  index  ( OPT/A211123) . 


0FT/A21113;  Calculate  simulation  indicator.  This  activity 
is  accomplished  by  taking  a  weighted  average  of  the  indices  for 
simulation  requirements  and  operating-cost  savings.  The  user  can 
use  the  default  weights  (6B6,  set  at  0.5  and  0.5)  for  this 
calculation  or  can  input  his  own  weights.  Since  the  weights 
represent  the  relative  importance  of  the  economic  and  non¬ 
economic  reasons  for  simulation,  they  are  subjective  and  require 
the  considered  judgment  of  the  user  for  their  assessment.  The 
equation  for  this  is: 
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ISIM^  =  WSR  *  ISRj  +  woes  *  lOCS^, 
where 

ISIMij.  =  the  simulation  indicator 

WSR  =  the  weight  for  simulation  requirements; 
woes  =  the  weight  for  operations-cost  savings; 

ISR.p  =  index  for  simulation  requirements; 
lOCS^  =  index  for  operations-cost  savings. 

This  equation  is  overridden  if  ISR.j  equals  1.0,  in  which  case  the 
simulation  indicator  will  be  set  to  1.0. 


117 


Simulation 

Requirements 


Smulation 

Requirements 

Indices 

OZiniA 


Task  Training 
Hours  and  Coats 
Learning 
Curves 

Baseline  Variable 
Operating 
Training  Device  Costs 


Operating'^  I 
Cost  Savings 
Indices 


Training  System  Data 
or  User  Judgement  Reqrnta/ 
Ops  Cost  Weights 
I  6 

Z. 


;  Calculate 

1  Simulation 

J  indicaiors 

T _ _ 

3 

Operabng 
CM  Savings 
021i12A 


Simulator 

Requirements 

Indcators 

02nA 


021112B 


Node: 

0PT/A2in 


Title:  Evaluate  Simulation  Indicators 


Date:  7/12/88 


Number 
OPT- 13 


OPT/A21112:  Calculate  Operating-Cost  Savings 

This  process  measures  the  validity  of  replacing  some  time  on 
the  actual  equipment  with  simulator  time  for  cost  reasons. 

OPT/A211121:  Minimize  total  operating  cost.  This  analysis 

uses  the  learning  curves  for  actual  equipment  and  for  an  existing 
FMS  (OID),  task  entry  and  standard  performance  levels  (lA), 
actual  equipment  (operational  and  non-operational )  hours  for 
training  without  the  simulator  (2A),  and  the  hourly  operating 
cost  of  the  FMS  (3A)  and  AE  (OlE)  to  compute  the  FMS  (0211121B) 
and  actual-equipment  (0211121A)  training  times  that  minimize  the 
total  operating  cost  required  to  train  each  task.  The  cost  of 
training  on  actual  equipment  assumes  that  operational  and 
non-operational  hours  are  reduced  in  constant  proportions  by  the 
simulator.  The  methods  described  by  Cronholm  (1985)  will  be  used 
as  the  mechanism  for  this  minimization.  Both  the  FMS  and  actual- 
equipment  learning  curves  are  power  functions,  and  the  transfer- 
of-training  function  is  assumed  linear  with  the  multiple  equal  to 
the  asymptote  of  the  device  from  which  transfer  is  occurring. 
Thus,  the  learning  curve  has  the  following  form: 

Pxi(t)  =  ASM.jj_  (1  -  [1  TM.j.iTSF.j.(HS.I.i  +  t)]“^}. 
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ASM 


Ti 


the  level  of  performance, 

the  training  time, 

time  scaling  factor 

the  effect  of  previous  experience, 

time  multiplier 

the  learning  rate  {assumed  to  be  0.7),  and 

the  asymptote  of  the  learning  curve,  a  measure  of 

transfer  of  training. 


Cronholm  (1985)  showed  that  the  derivative  of  training 
performance  on  the  AE  with  respect  to  training  time  on  the  FMS 
must  be  equal  to  the  FMS  cost  rate  divided  by  the  AE  cost  rate  to 
minimize  total  operating  cost.  Using  the  above  learning  curve, 
this  translates  to: 
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Where 

^FMS  “  l+TMp[^gTSF(HSpj^,5+tpj^g)  )  ~^FMS]  is  the 

FMS  learning  curve, 

Pae  =  ASMae  *  [l-(l+TMAETSF(HSAE+tAE) )“^AE]  is  the  AE 
learning  curve 

VHRCpf^jg  =  hourly  cost  of  FMS 
VHRCj  ae  ~  hourly  cost  of  AE 

The  subscript  T  indicating  the  task  is  assumed  throughout  this 
section.  tpj^g  is  solved  for  as  follows: 

^FMS  =  TMZ"!  TSF  "  P^/ASMpj^g)-^/^  -  (1  -  PENTRY/ASM 

FMS 

Where 


ENT  =  the  entry  level  performance  for  task  T,  and 
PX  =  the  crossover  performance  level  between  the  FMS  and  the 
AE ,  calculated  by  the  following  equation. 


PX 


ASM^e  -  (ASM^„3  »  z| 
1  -  z 


where 
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r/(r+l) 


t._  Is  calculated  as  follows: 

AE 

*AE  =  TH  ~  ISF  '<1  -  STD/ASM^g)-*^'-  -  (1  -  PX/ASM^^ ) ] 


OPT/A211122:  Calculate  cost  savings.  This  is  a 

straightforward  operation  using  the  results  of  the  previous 
process.  The  operating  cost  without  a  simulator  is  CVAR.p  (013C). 
This  minimal  cost  is  the  simulator  cost  (hours  times  operating 
cost)  plus  the  reduced  actual  equipment  operating  cost.  The  cost 
savings  (OCS^;  021112A),  then,  is  the  difference  between  the 
minimal  cost  and  the  cost  of  training  without  a  simulator. 

OPT/A211123:  Calculate  index  of  operating-cost  savings . 

This  analysis  uses  the  actual-equipment  cost  savings  of  all  tasks 
to  compute  a  normalized  index,  such  that  the  task  with  the 
greatest  operating-cost  savings  receives  an  index  value  of  1 : 
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lOCSj  =  OCS.j,/max(OCS.j.) 

T 

where 

lOCSj.  =  the  operating-cost  savings  index  for  task  T,  and 
OCS^  =  the  operating-cost  savings  for  task  T. 

This  index  is  guaranteed  to  be  between  0.0  and  1.0. 
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OPT/A2112;  Estimate  Development  Costs 

This  process  uses  the  task  cue  and  response  requirements  to 
estimate  the  development  cost  of  a  training  device  that  could 
provide  training  to  meet  the  performance  standard  for  task. 


OPT/A21121:  Calculate  Fidelity  Dimension  Costs.  This 

procedure  uses  a  technical  performance  index  to  estimate  a  cost 
equation  for  fidelity  dimension  j  using  the  following  equations: 


DCST.J.  j  ( PRQT.J.  j  ) 


=  CMIN.  +  (CMAX^  -  CMINJ 
J  J  j 


-d/Ej)  Ind  - 

■  FRQT^j 

-  TPMIN . 

-E 

(1  -  e  j)) 

TPMAX . 

-  TPMIN. 

L  J 

J  J 

where 


TPMINj  =  Technical  performance  of  the  least  capable  FMS  option 
for  cue/response  dimension  j 

TPMAXj  =  Technical  performance  of  the  most  capable  FMS  option 
for  cue/response  dimension  j 

CMINj  =  development  cost  of  the  minimum  technical  performance 
index  on  fidelity  dimension  j. 
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development  cost  of  the  maximum  technical  performance 
index  (set  to  1.0)  on  fidelity  dimension  j, 
empirically  determined  cost  equation  parameter  for 
fidelity  dimension  j, 

cue  and  response  requirement  for  task  T  on  fidelity 
dimension  j. 

If  FRQT^j  is  less  than  TPMINj,  the  performance  parameter 
associated  with  CMINj ,  then 

FRQT^j 

DCSTtj(PRQTtj)  =  55—  X  CMINj. 

OPT/A21122:  Sum  Across  Dimensions.  The  development  cost 

for  a  given  task  is  the  sum  of  the  fidelity-dimension  estimates. 
The  index  for  development  cost  estimates  is  the  normalization  of 
the  costs  so  that  the  task  with  the  highest  development  cost  has 
an  index  value  of  1.0  and  the  others  are  scaled  accordingly. 


CMAXj  = 
= 

FRQTtj  = 


DEVCj  =  I  DCSTjj 

J 

IDEVC-r  *  DEVC^/(raax(DEVCT)  ) 
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OPT/A212:  Develop  Simulation  Recommendations 

This  process  uses  the  task  indicators  for  simulation  and 
estimated  development  cost  indices  to  formulate  a  recommendation 
for  a  full-mission  simulator  (FMS),  one  or  more  part-mission 
simulators  (PMS),  or  use  of  actual  equipment  (AE). 

0PT/A2121:  Prepare  scatterplot  of  tasks.  This  is  a  process 

that  produces  an  x,y  plot  of  each  task;  that  is,  the  simulation 
requirement  indicator  (0211A)  is  plotted  along  the  y-axls  and  the 
development  cost  index  (021 IB)  is  plotted  along  the  x-axis  for 
each  task. 

OPT/A2122;  Calculate  simulator-option  s\immaries.  This 
analysis  uses  the  scatterplot  from  the  previous  process  to  assign 
tasks  to  recommendation  regions  and  then  compute  summaries  for 
the  FMS,  PMS,  and  AE  for  the  options  within  the  scope  of  this 
analysis.  In  the  three  operations  included  in  this  activity,  the 
cost  boundary  between  the  AE  region  and  the  FMS  and  PMS  regions 
is  determined  ( 0PT/A21221 ) ,  tasks  are  assigned  to  the  appropriate 
regions  (OPT/A21222 ) ,  and  summary  values  are  calculated  for  the 
three  regions  (OPT/A21223 ) . 
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The  option  summaries  that  result  from  this  process  Indicate 
the  relative  economic  and  training  Improvement-based  effects  that 
an  FMS  or  PMS  might  have.  If  these  numerical  summaries  are  high 
enough  for  either  or  both  of  the  FMS  or  PMS  options,  then  the 
OSBATS  recommendation  will  be  favorable. 

OPT/A21221;  Compute  cost  boundary.  This  process  uses  data 
about  operating-cost  savings  of  simulation  (021112A1),  the  life 
cycle  of  the  simulator  (3A4),  and  the  weight  given  operating  cost 
savings  Index  (6B6)  In  determining  the  Indicator  for  simulation 
requirements  to  determine  the  cost  trade-off  for  recommending  a 
simulator.  The  major  assumption  made  in  computing  this  cost 
boundary  Is  that  if  a  given  taslc  has  no  requirements  for 
simulation,  the  decision  to  use  a  simulator  is  based  on 
economics  only  (operating-cost  savings  versus  development  cost). 
Simulation  requirements  provide  additional  benefits  of  simulation 
that  are  added  to  the  economic  benefits.  The  following  three 
equations  are  the  basis  of  this  computation: 

(1)  ISIM  =  IOCS  *  woes  by  assumption 

by  definition 


(2)  IOCS  =  OCS  /  OCS 
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(3)  DEVC  =  OCS^ax  *  *  GRADS  cost  accounting 

where 


IOCS 

OCS 

ISIM 

woes 

DEVC 

PB 

TSIM 

GRADS 


index  of  operations  cost  savings; 

operations-cost  saving  of  the  simulator  {021112A); 
indicator  for  simulation; 

weight  for  operations  cost  savings  in  calculation  of 
ISIM  (6B6); 

development  cost  of  a  simulator; 
payback  period  needed  for  the  simulator  (3A4); 
number  of  tasks  over  which  the  simulator  development 
cost  should  be  amortized; 

the  required  number  of  graduates  per  year  (6A1). 


The  first  equation  embodies  the  assumption  that  there  are  no 
requirements  for  simulation.  The  second  equation  is  the 
definition  of  the  operating-cost  savings  index.  The  third 
equation  expresses  the  criterion  for  the  cost  boundary  that  the 
training  device  development  cost  is  saved  over  the  payback  period 
(3A4)  for  all  tasks  for  which  the  device  is  used  (tasks  with 
positive  cost  savings).  By  solving  for  ISIM  as  a  function  of 
DEVC  we  find 


ISIM  =  WOCS/(PB  *  TSIM  •  OGS^^j^  *  GRADS)  *  DEVC 
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However,  since  (TSIM  *  OCS-gjj)  overapproximates  the  savings 
due  to  the  simulator,  we  are  substituting  (from  OPT/A211122 ) : 

2]  ocSip< 

T 

So  we  have  a  linear  cost  boundary  with  slope, 

WOCS/(PB  *  I  OCSij.  *  GRADS), 

T 

relating  the  x  and  y  dimensions  of  the  scatterplot.  Any  task 
lying  above  this  line  would  qualify  for  a  simulator;  and  those 
below  would  not.  User  adjustments  in  the  cost  boundary  are 
allowed  due  to  the  uncertainty  in  the  actual  cost  savings  that 
would  result  from  simulation. 

OPT/A21222:  Assign  tasks  to  FMS ,  PMS  and  NoS  options .  The 

linear  cost  boundary  and  a  vertical  boundary  for  a  specified 
value  of  the  fidelity  index  that  separates  a  FMS  from  PMSs  are 
used  to  assign  tasks  in  the  scatterplot  to  FMS,  PMS  or  AE 
options.  (See  the  figure  opposite.) 

0PT/A21223:  Sum  task  weights  for  FMS,  PMS,  and  NoS 

options .  The  task  weights  (OlC)  of  all  tasks  in  the  FMS  region 
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*  FMS  •  Fun  Mission  Simulator,  PMS  -  Part-Mission  Simulator,  AE  -  Actual  Equipment 


Node: 

Title;  Calculate  Simulator  Option 

Date;  7/12/88  1  Numoer.  ! 

OPT/AS122 

Summaries 

i  OPT-.17  1 

of  the  scatterplot  are  sununed  to  be  the  FMS  summary.  The  same  is 
done  for  the  PMS  tasks  and  the  AE  tasks.  These  summary  values 
are  displayed  to  the  user. 

Simulation  Configuration  Module 


Cnpblesl  XapraMOUtton  at  Inults 
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VEtrrmw 


,  Number 
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OPT/A3;  Design  training  devices. 

Two  tools  are  used  to  aid  in  the  design  of  training 
devices.  The  first  tool  determines  the  cost-effective 
instructional  features  to  be  incorporated  into  a  training 
device.  The  second  tool  determines  the  device  areas  in  which 
technical  sophistication  and  fidelity  would  be  cost-effective. 

0PT/A31;  Select  instructional  features.  This  tool  examines 
the  instructional  features  that  would  improve  training  efficiency 
for  each  task  (OlA) .  Then  the  tool  identifies  the  instructional 
features  that  have  the  greatest  expected  impact  on  training 
efficiency,  given  their  cost.  The  output  of  this  module  is  a 
list  of  the  instructional  features,  ordered  by  decreasing 
benefit/cost  ratio.  Features  at  the  beginning  of  the  list 
provide  a  better  value,  given  their  development  cost,  than  the 
features  at  the  end  of  the  list.  The  optimal  set  of 
instructional  features  at  any  budget  may  be  determined  by 
selecting  features  in  order  of  decreasing  benefit/cost  ratio 
until  the  budget  is  met.  The  results  of  this  model  may  be 
Integrated  into  the  analysis  performed  by  the  fidelity 
optimization  model. 

0PT/A32:  Optimize  device  fidelity.  This  tool  addresses  the 

problem  of  how  much  should  be  invested  in  the  technical 
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sophistication  and  fidelity  of  a  training  device.  The  tool 
considers  several  dimensions  of  fidelity  that  are  related  to  task 
cue  and  response  requirements.  A  training  device  may  be  more  or 
less  sophisticated  on  each  fidelity  dimension.  The  module 
compares  task  requirements  on  these  dimensions  with  the  cost  of 
meeting  these  requirements,  in  order  to  determine  the  areas  in 
which  an  investment  in  increased  fidelity  can  be  justified  by 
increased  training  effectiveness.  The  output  of  this  model  is  a 
collection  of  training  device  designs  that  optimize  training 
effectiveness  as  a  function  of  Investment  cost.  Given  this  set, 
the  optimal  training-device  design  at  any  development  cost  may  be 
determined.  The  choice  of  the  single  training  device  design  that 
meets  training  criteria  at  the  lowest  cost  from  the  set  of 
optimal  device  designs  should  be  made  by  either  the  Training 
Device  Selection  Module  or  the  Resource  Allocation  Module. 
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OPT/A31;  Select  Instructional  Features 

This  tool  for  designing  training  devices  identifies,  given  a 
development  cost,  the  instructional  features  that  will  improve 
training  for  the  most  tasks.  Instructional  features  are  presumed 
to  Influence  the  efficiency  of  training  rather  than  the  maximum 
effect  of  training  or  transfer  of  training.  Thus,  instructional 
features  influence  the  time  multiplier  in  the  learning  model 
rather  than  the  asymptote.  Three  major  activities  compose  this 
tool:  candidate  features  are  selected  from  a  master  list,  the 

benefits  of  these  features  are  determined,  and  the  features  that 
provide  the  greatest  benefit  for  the  cost  are  determined.  The 
outputs  of  this  module  may  serve  as  inputs  to  the  Fidelity 
Optimization  Module. 

0PT/A311:  Select  tasks  and  candidate  features.  In  this 

activity,  the  user  selects  the  training  tasks  to  be  used  as  the 
basis  for  the  evaluation  (0311B).  This  selection  is  described  in 
node  0PT/A3111.  The  selected  tasks  may  be  one  of  the  clusters 
produced  by  the  Simulation  Configuration  Module  (02A),  the  tasks 
trained  on  one  device  as  determined  by  the  Training  Device 
Selection  Module  (04A),  or  a  group  of  tasks  selected  by  the  user. 
In  addition,  the  user  selects  the  instructional  features  to  be 
evaluated  in  the  analysis  (0311A)  from  a  master  list  of 
instructional  features  (5B).  This  selection  is  described  in 
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node  0PT/A3112.  Specific  procedures  for  this  activity  are 
described  in  the  detailed  diagram  for  this  node. 

OPT/A312:  Calculate  benefits  of  features.  In  this 

activity,  the  benefits  of  the  candidate  instructional  features 
{0312A)  are  determined.  In  order  to  determine  instructional 
feature  benefit,  the  module  must  aggregate  (1)  the  number  of 
tasks  for  which  the  instructional  feature  is  relevant  (OlA),  (2) 
the  economic  costs  of  training  these  tasks  on  actual  equipment 
(OlC),  and  (3)  the  likelihood  that  the  instructional  features 
will  be  used  by  instructors  (5B2).  The  overall  benefit  of  an 
instructional  feature  is  the  product  of  these  three  factors.  The 
subactivities  of  this  activity  first  calculate  a  measure  of 
feature  effectiveness  that  incorporates  the  first  two  factors 
described  above  (0PT/A3121),  and  then  incorporate  the  third 
factor  (OPT/A3122). 

0PT/A313:  Select  optimal  order  of  features.  In  this 

activity,  the  instructional-feature  benefit  measures  (0312A)  are 
combined  with  assessments  of  feature-investment  cost  (5B1),  and 
the  ratio  of  benefit  to  cost  is  used  to  determine  the  optimal 
selection  of  instructional  features  at  any  cost  (03A1).  The  user 
may  examine  the  optimal  selection  of  features  at  any  development 
cost.  In  addition,  the  entire  ordered  list  of  instrur  nal 
features,  or  some  portion  of  it,  may  serve  as  input  t<-  t' e 
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fidelity  optimization  module.  The  optimal  order  of  features  is 
determined  by  computing  the  benefit/cost  ratio  for  each  feature 
(OPT/A3131),  sorting  the  features  by  benefit/cost  ratio 
{OPT/A3132),  and  listing  the  instructional  features  (0PT/A3133). 
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0PT/A311;  Select  Tasks  and  Candidate  Features 


The  data  base  contains  a  master  list  of  instructional 
features.  The  user  may  not  want  to  consider  all  features  on  the 
list.  Consequently,  this  activity  gives  the  user  an  opportunity 
to  modify  the  list  of  instructional  features.  In  addition,  the 
user  is  given  an  opportunity  to  select  the  tasks  that  are  to  be 
used  to  evaluate  the  candidate  instructional  features. 

0PT/A3111;  Select  device  tasks.  The  tasks  included  in  the 
analysis  define  the  requirements  for  the  training  device  being 
designed.  The  optimal  instructional  features  for  the  training 
device  depend  on  these  training  requirements.  The  Simulation 
Configuration  Module  (02A)  and  Training  Device  Selection  module 
{04A)  both  make  recommendations  regarding  the  tasks  assigned  to  a 
training  device.  The  user  may  use  the  recommendations  of  one  of 
these  models,  or  may  select  other  tasks  to  be  used  in  determining 
instructional-feature  benefit. 

0PT/A3112:  Select  instructional  features.  In  this  step', 

the  user  selects  the  instructional  features  to  be  evaluated  in 
the  analysis  from  a  master  list  of  possible  features  (SB).  In 
this  way,  the  user  may  eliminate  from  consideration  those 
features  that  are  infeasible  for  reasons  outside  the  scope  of  the 
model . 
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OPT/A312r  Calculate  Benefits  of  Features 

An  Instructional  feature's  benefit  depends  primarily  on  the 
number  of  tasks  for  which  the  feature  enhances  training 
efficiency.  In  aggregating  over  tasks,  tasks  that  are  expensive 
to  train  on  actual  equipment  are  given  higher  weight  than  tasks 
that  are  relatively  inexpensive  to  train.  The  task  weight  (OIC) 
is  a  measure  of  the  costs  to  train  the  tasks  on  actual  equipment. 
A  feature's  overall  effectiveness  measure  is  multiplied  by  a 
benefit  weight  that  represents  the  likelihood  that  a  feature 
would  actually  be  used  by  an  instructor. 

OPT/A3121:  Calculate  effectiveness  score.  The 

effectiveness  score  (EFFj^;  03121A)  aggregates  the  feature/ task 
match  matrix  (IFTmj^)  over  tasks.  The  overall  score  is  a  weighted 
sum  of  IFT^jj  (OlA),  with  weights  being  the  economic  weights 
(TWT^)  of  the  tasks  (OIC).  The  effectiveness  score  (EFFj^)  is 
given  by  the  following  equation: 

EFFj^  =  E  (IFT.j.j^  X  TWT.J.) 

T 

OPT/A3122:  Calculate  feature  benefit.  Recent  analyses 

(Logicon,  1985)  suggest  that  an  evaluation  of  Instructional 
features  consider  overall  feature  benefit  in  addition  to  the 
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task— sped fic  measures  (03121A)  described  above.  This  benefit 
accounts  for  frequency  of  need,  instructor  loading,  and  feature 
useability.  We  have  implemented  these  suggestions  by 
incorporating  an  instructional-feature  benefit  weight  (IFWTj^). 
The  benefit  weight  ( 5B2 )  summarizes  historical  data  on  the 
likelihood  that  an  instructional  feature  would  be  used.  The 
overall  benefit  (BENj^)  for  instructional  feature  k  is  given  by: 

BENj^  =  E®’^k  ^  IFWTj^ 
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OPT/A313;  Select  Optimal  Features 

This  process  combines  the  calculated  instructional-feature 
benefits  (0312A)  with  assessed  feature  costs  (5B1).  It 
determines  the  optimal  features  to  be  included  in  the  design  of  a 
training  device  as  a  function  of  the  investment  cost  budget  for 
instructional  features.  The  user  may  select  one  or  more  options 
to  be  evaluated  in  the  Fidelity  Optimization  Module. 

0PT/A3131:  Compute  benefit/cost  ratios  for  options.  The 

k-th  instructional  feature  is  represented  by  both  a  benefit 
(BENj^;  0312A)  and  a  development  cost  (IFCj^;  5B1 )  .  This  activity 
divides  the  benefit  by  the  cost  to  obtain  a  benefit/cost  ratio 
(BENj^/IFCi^)  . 

0PT/A3132:  Sort  options  by  benefit /cost  ratio.  This 

activity  orders  the  features  according  to  decreasing  benefit/cost 
ratio.  If  the  features  being  considered  are  presumed 
independent,  they  can  be  properly  prioritized  on  the  basis  of 
BENj^/IFCj^.  The  assumption  of  independence  deserves  some 
Investigation,  particularly  in  terms  of  its  interaction  with  the 
assumptions  used  to  estimate  the  effect  of  instructional  features 
in  the  Training  Device  Selection  Module. 
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OPT/A3133;  List  optimal  features.  This  activity  presents 
the  results  of  the  analysis  so  that  the  user  may  determine  the 
instructional  features  that  should  be  included  to  meet  cost  or 
benefit  criteria.  The  optimal  list  of  features  at  any  cost  (or 
benefit)  level  is  found  by  adding  features  to  the  list  in  sorted 
order  until  the  cumulative  cost  (or  benefit)  is  equal  to  the 
criterion  level.  This  activity  takes  the  range  of  costs  and 
produces  as  output  the  set  of  optimal  instructional-feature 
packages  for  all  costs  in  the  range.  Included  in  the  output  are 
the  cost  and  benefit  for  each  package. 
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The  goal  of  this  tool  is  to  consolidate  task  cue  and 
response  requirements  for  the  tasks  being  considered  for  training 
on  a  training  device  with  training  device  fidelity  options  so 
that  optimal  devif^e  designs  can  be  identified. 


0PT/A321:  Construct  training  device  options.  This  activity 

allows  the  user  to  examine  the  task  cue  and  response  requirements 
(OIB)  of  the  tasks  specified  in  the  training  concept  (02A)  to 
select  relevant  fidelity  dimensions  (in  OPT/A3211)  and  pick  the 
minimvun  and  maximum  options  in  each  fidelity  dimension  (in 
0PT/A3212)  from  a  master  list  of  options  in  the  data  base  (4A). 
The  user  is  also  ashsd  to  determine  whether  the  results  of  the 
instructional  features  module,  if  available,  should  be 
Incorporated  as  a  fidelity  dimension  in  this  module  (0PT/A3213). 
Finally  the  user  may  discard  any  options  between  the  minimum  and 
maximum  for  a  given  fidelity  dimension  (0PT/A3214).  These 
options  and  dimensions  have  been  designed  to  be  as  independent  of 
each  other  as  possible — independent  in  the  sense  that  cost  and 
benefit  of  each  option  do  not  depend  on  the  options  chosen  for 
other  dimensions.  The  outputs  of  this  activity  are  a  set  of 
candidate  training  device  options  (0321A)  and  associated 
technical  performance  indices  (0321B)  from  the  data  base  (4A3). 
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OPT/A322:  Calculate  costs  and  benefits  of  options.  This 

process  has  four  operations  that  establish  development  costs  and 
training  benefits  for  each  option.  These  costs  and  benefits  are 
comparable  across  all  fidelity  dimensions  and  are  based  upon  the 
technical  performance  indices  for  each  option.  Costs  are 
determined  from  technical  performance  by  a  logarithmic  estimation 
function  in  the  first  subactivity  (OPT/A3221).  The  remaining 
three  subactivities  determine  benefit  by  determining  task 
trainability  (OPT/A3222),  calculating  a  benefit  score  within 
each  fidelity  dimension  (0PT/A3223),  and  determining  weights  that 
place  benefit  on  a  common  scale  across  dimensions  (0PT/A3224). 

0PT/A323:  Compute  optimal  device  designs.  This  process  has 

four  operations.  The  optimal  training  device  designs  are  based 
upon  the  incremental  benefit-to-cost  ratios  of  the  options. 

After  these  ratios  are  computed  (OPT/A3231)  the  options  can  be 
sorted  in  priority  order  (0PT/A3232),  and  optimal  designs  can  be 
defined  for  user-specified  cost  or  performance  levels 
(0PT/A3233).  Alternative  optimal  device  designs  may  then  be 
selected  (0PT/A3234)  to  be  compared  by  the  Training  Device 
Selection  or  Resource  Allocation  Module  to  determine  vt'nich  meet 
the  training  requirement  at  the  lowest  cost.  The  specific 
procedure  for  this  activity  is  described  in  the  detailed 
description  for  this  node. 
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OPT/A321:  Construct  Training  Device  Options 

This  activity  develops  the  building  blocks  that  describe 
options  within  each  of  the  fidelity  dimensions.  The  user  may 
examine  the  master  list  of  fidelity  dimensions  and  eliminate 
those  that  are  inappropriate,  set  the  limits  of  the  options 
considered  by  the  analysis,  and  eliminate  specific  options  that 
are  infeasible  for  reasons  outside  the  scope  of  the  analysis. 

OPT/A32I1;  Select  tasks  and  relevant  dimensions .  The  tasks 
to  be  trained  on  the  subject  training  device  are  specified  by 
either  the  Simulation  Configuration  Module  (02A)  or  the  Training- 
Device  Selection  Module  (04A1).  The  user  may  modify  the  outputs 
of  these  models,  or  may  select  other  tasks  to  be  used  as  the 
basis  for  the  evaluation.  In  addition,  the  user  may  choose  to 
eliminate  some  fidelity  dimensions  (4A)  from  consideration  in  the 
analysis.  The  output  of  this  activity  is  a  set  of  relevant  tasks 
(03211A)  and  dimensions  (03211B)  that  will  form  the  basis  of  the 
analysis . 

0PT/A3212:  Set  minimum  and  maximum  options.  This  process 

allows  the  user  to  compare  the  options  (4A)  in  each  relevant 
fidelity  dimension  (03211B)  to  the  task  cue  and  response 
requirements  (OIB)  on  each  dimension  for  the  selected  tasks 
(03211A}.  This  comparison  should  lead  the  user  to  either  accept 
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the  minimum  and  maximum  options  of  each  dimension  contained  in 
the  OSBATS  data  base  or  specify  his  own,  less-severe  minima 
and/or  maxima.  The  output  of  this  process  (03212A)  defines  the 
range  of  training  device  designs  that  will  be  considered  by  the 
analysis . 

OPT/A3213:  Incorporate  instructional-features  options.  This 

activity  asks  the  user  to  decide  whether  the  results  of  the 
instructional  features  model  (03A1),  if  available,  should  be 
incorporated  as  a  fidelity  dimension.  If  the  user  decides  to 
incorporate  instructional  features,  the  costs  and  benefits  of 
these  options  are  Included  in  the  fidelity  optimization  module. 
The  benefits  of  these  options  are  computed  in  0PT/A3224. 

0PT/A3214:  Define  intermediate  options.  This  process 

allows  the  user  to  discard  specific  intermediate  options  of  any 
fidelity  dimension  (4A2).  The  outputs  of  this  process,  which 
guide  and  drive  the  remainder  of  the  analysis,  are  (1)  the  set  of 
options  to  be  evaluated  in  the  analysis  (0321A)  and  (2)  their 
associated  technical  performance  indices  (0321B;  from  4A3). 
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OPT/A322:  Calculate  Costs  and  Benefits  of  Options 

This  process  yields  the  benefit  and  cost  data  needed  to 
compute  the  incremental  benefit-to-cost  ratios  for  the 
optimization  in  OPT/A323. 

OPT/A3221;  Calculate  costs.  This  activity  addresses  the 
development  cost  of  a  training  device  and  is  a  straightforward 
calculation  from  fidelity  dimension  data  (4B)  using  the  cost 
equations  shown  below.  The  cost  equations  relate  the  development 
cost  to  the  technical  performance  index  in  the  range  of  options 
under  consideration,  as  follows: 

CDEVj„(TPj^)  =  CMINj  +  (CMAXj.  -  CMINj)  *  CRELj„,  *  TPRELj^, 
where 

CDEV'jjjj  =  the  development  cost  of  training  device  option  m  on 
fidelity  dimension  j, 

CMINj  =  the  development  cost  of  the  lease  capable  training 
device  option  on  fidelity  dimension  j, 

CMAXj  =  the  development  cost  of  the  most  capable  training 
device  option  on  fidelity  dimension  j,  and 
CRELj^  =  the  normalized  development  cost  of  training  device 
option  m  on  fidelity  dimension  j. 
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The  relative  technical  performance  (TPRELj-)  is  also  defined  on  a 
scale  from  0  to  1.0  as  a  function  of  the  absolute  technical 
performance  (TP),  as  follows: 

TP._  -  TPMIN. 

TPREL^  (TPl  =  - - J_ 

TPMAXj  -  TPMIN j 

where 

TPjni  =  the  technical  performance  index  of  training  device 
option  m  on  fidelity  dimension  j. 

TPMAXj  =  the  technical  performance  of  the  most  capable  option  on 
fidelity  dimension  j, 

TPMIN j  =  the  technical  performance  of  the  least  capable  option 
on  fidelity  dimension  j,  and 
TPRELjni  =  the  normalized  (between  0  and  1.0)  technical 

performance  of  training  device  option  m  on  fidelity 
dimension  J. 

We  estimate  TPRELjjjj  from  the  cost  using  the  following  equation: 
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^  1  -  e<-^1 


1  -  e-=J 


where 

Ej  =  a  constant  that  describes  the  fidelity  dimension  (4B3). 

To  compute  the  costs,  the  inverse  of  this  function  would  be 
computed  using  the  following  equation: 

ln[l  -  TPRELjj^d  -  e~^j)] 


This  function  has  the  property  that  technology  becomes  more 
costly  as  the  constant  exponent  (Ej)  increases  in  size. 

OPT/A3222:  Determine  task  tralnability.  This  is  the  first 

step  in  the  calculation  of  benefits  for  training  device  options. 
The  output  variable  for  this  process  (TRN.j.j^)  is  the  intersection 
of  task  T  and  option  m  of  fidelity  dimension  j.  For  each  such 
intersection  being  considered,  the  task  cue  and  response 
requirement  (OIB)  is  compared  to  the  technical  performance  index 
(0321B)  of  the  option.  If  the  technical  performance  index  of  the 
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option  is  greater  than  or  equal  to  the  requirement  of  the  task, 
TRNTjjj,  is  set  equal  to  1.0.  Otherwise,  it  is  set  equal  to  0.0. 
This  determination  is  undertaken  for  all  tasks  and  all  options 
under  consideration. 

0PT/A3223;  Calculate  benefit  scores.  This  process  uses  the 
variable  TRNjj^  {03222A)  and  the  task  weights  (OIC)  calculated  in 
0PT/A12  to  calculate  the  benefit  score  of  each  option.  The 
benefit  score  is  defined  to  represent  the  relative  benefit  of  an 
option  within  its  fidelity  dimension.  For  all  options  within  a 
fidelity  dimension  the  benefit  scores  are  normalized  to  fall 
between  0  and  1.0;  that  is,  the  least  effective  (or  minimum) 
option  is  defined  to  have  a  benefit  of  0  and  the  most  effective 
(or  maximum)  option  a  benefit  of  1.0.  This  normalization  defines 
the  benefit  to  be  total  benefit  above  the  minimum  fidelity  option 
in  the  dimension  and  facilitates  the  development  of  benefit 
weights  later. 

First,  the  non-normalized  benefit  scores  of  the  options 
(UBENjjjj)  are  calculated  as  follows: 

UBENj„  =  I  TRNtj^  TWTt 
T 
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where  TWT^  is  the  economic  weight  of  task  T.  Now  the  normalized 
benefit  scores  of  the  options  (NBENjj^)  can  be  calculated  as 
follows : 

NBENjn  *  (UBENj„  -  UBENj ^ ) / ( UBEN -  UBENjj), 
where 

UBENjj^  =  the  non-normal ized  benefit  of  the  minimum  option  of 
fidelity  dimension  j,  and 

UBENj{^  =  the  non-normal  ized  benefit  of  the  maximum  option  of 
fidelity  dimension  j. 

Due  to  the  mathematical  operations  involved  we  are  assured  that 
NBENu,^  does  not  decrease  as  m  increases  for  a  fixed  j.  If  NBENj^ 
and  NBENj^+1  are  equal,  then  the  m+1  option  of  dimension  j  should 
be  dropped  from  consideration. 

0PT/A3224:  Calculate  fidelity  benefit  weights.  This 

activity  uses  the  task  cue  and  response  requirement  (OIB)  and  the 
task  weights  (OIC)  to  calculate  a  set  of  fidelity-dimension 
weights  that  place  benefit  on  a  common  scale  across  fidelity 
dimensions.  First,  a  modified  task  cue  and  response  requirement, 
RM^j,  is  computed  as  follows: 


152 


Training  Oevica 
Options 


Fidelity 
Dimension 
Costs _ 


4S 


0321A 

K 

_ 


Selected  Instnictional 
Features  — 

Tectinieal  — t 
Performance 
lndicas0321B 


Calculate 

Costs 


Task  Cue  i  Response 
Requirements 

01B 


Task 

TramabiUty 


Tasks 
Trained  By 

I  )  /  03222A 


EcoiMmic  Task  W«ghts  - 


01C 


T 


Option 


Scores 

0322A3 


Calculate 

Benefit 

Scores 


/ 


Option 

Costs 

0322A1 


_f _ 


Fidelity 

Dimension 

Weignts 

_  \  0322A3 


Calculate 

Benefit 

Waignta 


-e~ 


1  Node: 

1  TiM:  Calculate  Costa  and  Benefits  of 

1  Date;  7/12/88 

Number 

1  OPT/A322  1 

1  Options 

1 

OPT-25 

RM^j  =  (FRQT^j  -  TPji)/(TPjm  -  TPj^^) 
where 

FRQTipj  =  is  the  task  cue  and  response  requirement  for  task  T  and 
fidelity  dimension  j  (OlB), 

TPu^  “  'the  technical  performance  index  of  the  first  level  for 
fidelity  dimension  j  {0321B), 

TPjj^  s*  the  Technical  Performance  index  of  the  highest  level 
for  fidelity  dimension  J  (0321B) . 

This  modification  of  the  task  cue  and  response  requirement 
normalizes  the  requirement  so  that  it  is  zero  for  the  least 
capable  option  for  each  fidelity  dimension,  and  one  for  most 
capable  option.  Note  that  values  less  than  zero  should  be  set  to 
zero.  The  benefit  weights  for  each  fidelity  dimension  are  the  sum 
(across  all  tasks)  of  the  modified  task  cue  and  response 
requirement  times  the  economic  weights.  That  is, 


BWTj 


L  TWT^ 


These  fidelity  benefit  weights  are  then  normalized  to  sum  to  one 
to  raaint-’.n  scale  relationships  among  benefit  values. 
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A  single  value  that  incorporates  the  benefit  value  and  the 
benefit  weight  for  instructional  features  is  calculated  based  on 
the  relative  costs  of  instructional  features  compared  to  the 
fidelity  dimensions.  If  F-  is  defined  as  the  set  of 
instructional  features  included  in  option  ra  of  the  instructional 
feature  dimension,  and  is  the  set  of  instructional  features  in 
the  maximum  option,  then: 


L  BEN  ■ 

Z  IFC  ■ 

keF 

m 

Z  BEN. 

Z  CMAX. 

•  100 


The  first  term  computes  the  fraction  of  the  benefits  associated 
with  the  instructional  features  (BENj^)  included  in  option  m 
compared  to  the  maximum  option,  M.  The  second  term  computes  the 
ratio  of  the  maximum  cost  of  the  instructional  features  being 
considered  (IFCj^)  to  all  of  the  costs  associated  with  the 
fidelity  dimensions  (CMAXj).  The  100  is  used  to  place  all  of  the 
weighted  scores  on  the  same  scale.  Thus,  no  further  weight  is 
applied  to  the  instructional  feature  variable  (BWTjp  =  i). 
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OPTA323;  Select  Optimal  Device  Designs 

This  process  assembles  the  device  fidelity-dimension  levels 
based  upon  the  incremental  benefit-to-cost  ratios  to  attain  these 
levels.  It  uses  this  information  to  determine  the  optimal 
training  device  design  at  any  user-specified  level  of  development 
cost  or  benefit.  These  designs  may  be  evaluated  further  by  using 
them  as  inputs  to  the  Training-Device  Selection  Module  or 
Resource  Allocation  Module,  to  determine  which  option  meets  the 
training  requirements  at  the  least  cost. 

0PT/A3231 :  Compute  benefit-to-cost  ratios  for  the  options. 

This  activity  uses  the  cost  and  benefit  data  from  the  previous 
major  process  (0322A).  The  weighted  incremental  benefit-to-cost 
ratio  of  option  ra  from  fidelity  dimension  j,  (B/C)j^,  is  defined 
as  follows: 

(B/C)j„  .  BWTj  •  INBENj„  -  NBENj^.j ) / I CDEVj„  -  CDEVj^.i), 

Where 

CDEVjJJ^  =  the  development  cost  of  option  m  in  dimension  j. 

Before  the  options  can  be  sorted  by  their  benefit-to-cost  ratios, 
these  ratios  must  be  guaranteed  to  be  decreasing  within  each 
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fidelity  dimension.  That  is,  must  be  greater  than 

This  is  done  by  dropping  any  level  m  from 
consideration  that  does  not  meet  this  test  of  decreasing 
benefit/cost  ratio  and  setting  equal  to 

BWTj  •  (NBENj„+l  -  NBENj„.i)/CDEVj„^l  -  ODEVj„.i ) . 

This  recomputation  must  continue  until  all  levels  within  a 
fidelity  dimension  that  have  not  been  dropped  satisfy  this  test. 

OPT/A3232:  Sort  options  by  benefit-to-cost  ratios.  Because 

these  options  have  been  constructed  to  be  independent  building 
blocks  of  a  training  device  (to  the  extent  that  this  is 
possible) ,  they  can  be  prioritized  solely  on  the  basis  of  the 
incremental  (B/C)j„.  The  output  of  this  activity  (03232A)  lists 
the  options  on  all  fidelity  dimensions  in  order  of  decreasing 

0PT/A3233:  Compute  device  designs.  This  process  takes 

input  from  the  user  as  to  the  target  cost  or  benefit  levels  of 
device  designs  that  need  to  be  considered.  Associated  with  the 
prioritized  list  of  options  will  also  be  the  appropriate 
weighted  incremental  benefit, 

(BWTj  •  (NBENjn  -  NBENjn,_i)), 
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and  incremental  cost, 

(CDEVj„  -  CDEVj^.i), 

of  each  option.  These  incremental  costs  and  benefits  can  be 
accximulated  down  the  priority  list  to  determine  the  total  cost 
and  benefit  of  a  device  design  that  contains  the  given  option  and 
all  higher  priority  options  above  it.  At  any  time  the  user  can 
re-enter  this  model  and  request  new  device  designs  by  giving  new 
cost  or  benefit  levels.  The  user  may  select  one  or  more  of  the 
optimal  device  designs  to  accept  or  to  evaluate  further  using  the 
Training  Device  Selection  Module  or  Resource  Allocation  Module. 

0PT/A3234:  Calculate  cost  elements.  The  costs  of  the 

optimal  training  devices  selected  by  the  user  represent 
development  cost  only.  In  this  activity,  the  development  costs 
are  accumulated,  and  the  other  cost  elements  arc  estimated.  The 
total  investment  cost  (INVji^)  for  the  device  is  simply  the  sum  of 
the  CDEVjjj,  for  the  selected  levels  of  each  fidelity  dimersion. 

The  annual  fixed  operating  cost  {POC_j)  and  hourly  variable 
operating  cost  (VHRC^)  are  derived  from  INV^,  as  follows: 

FOC^  =  INV^  /  FCFAC 
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VHRCi  =  INV^  /  VCFAC 
where 

FCFAC  =  a  constant  fixed-cost  estimation  factor  (6B8),  and 
VCFAC  =  a  constant  variable-cost  estimation  factor  ( 6B9 ) . 

Device  utilization  (UMAX^)  and  life  cycle  (LC^)  are  set  to  the 
constant  values,  UEST  (6B10)  and  LCEST  (6B11),  respectively. 
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0PT/A4 :  Assign  Training  to  Devices 

The  goal  of  this  activity  is  to  assign  training  on  each  task 
among  previously  defined  training  devices.  These  training 
devices  may  have  been  designed  by  the  modules  in  0PT/A3,  or  they 
may  be  templates  of  existing  devices.  In  either  case,  the 
activity  must  determine  the  allocation  scheme  that  meets  the 
training  requirements  at  the  minimum  cost.  Two  tools  were 
designed  to  address  this  problem.  The  first  tool  makes 
simplifying  assumptions  to  allow  interactive  operation.  For 
example,  this  tool  assumes  that  as  many  devices  as  needed  may  be 
acquired  to  meet  training  time  demands.  The  second  tool  makes 
fewer  assumptions,  and  allows  the  user  to  impose  additional 
constraints  on  training  devices  usage.  For  example,  it  considers 
the  discrete  costs  of  device  acquisition,  and  it  allows  the  user 
to  place  a  ceiling  on  the  number  of  devices  of  each  particular 
type  that  may  be  acquired. 

0PT/A41:  Select  training  devices  for  tasks.  This  tool,  the 

Training  Device  Selection  Module,  determines  a  group  of  devices 
that  should  be  used  to  train  each  task,  and  calculates  the  amount 
of  time  a  student  should  spend  training  on  each  of  these  devices. 
The  chosen  training  devices  are  conditionally  optimal,  in  that 
they  meet  the  training  requirements  at  the  minimum  cost  subject 
to  the  simplifying  assumptions  mentioned  above.  Those 
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assumptions  imply  that  training  device  cost  functions  are  simple 
linear  functions  of  per-student  training  device  use  (time).  This 
simplification  allows  the  module  to  perform  the  optimization  in 
seconds . 

0PT/A42:  Allocate  training  resources  to  training  devices. 

This  tool,  the  Resource  Allocation  Module,  also  determines 
device  usage  to  minimize  total  training  costs,  but  treats  the 
cost  relations  in  more  detail  and  allows  the  imposition  of 
certain  constraints.  In  particular,  it  weakens  the  assumption  of 
simple  linear  cost  functions  and  allows  for  the  inclusion  of 
assets  on  hand.  The  new  form  for  a  training  device  cost  function 
is  assumed  to  be  piecewise  linear,  with  (possibly)  an  initial 
segment  corresponding  to  assets  on  hand,  and  the  subsequent 
segments  corresponding  to  acquisition  of  another  one  of  that 
device.  The  optimization  conducted  in  this  module  considers 
constraints  that  limit  the  amount  of  time  a  training  device  may 
be  used  or  the  performance  level  at  which  the  device  may  be  used. 
As  a  result  of  the  increased  generality,  the  Resource  Allocation 
Module  will  require  significantly  greater  time  for  solution  than 
the  Training  Device  Selection  Module.  Consequently,  its  use  may 
be  more  appropriate  in  a  "batch"  mode  than  in  an  interactive 
mode.  Provision  is  made,  however,  to  speed  the  algorithm  by 
sacrificing  some  of  the  detailing  in  cost  function 
representation . 
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OPT/A41;  Select  Training  Devices  for  Tasks 

This  tool  applies  cost-effectiveness  analysis  to  select  the 
training  devices  that  meet  training  requirements  at  the  minimum 
cost.  The  model  considers  (1)  the  fidelity  of  each  candidate 
training  device,  (2)  the  instructional  features  present  in  each 
candidate  training  device  that  can  reduce  the  time  required  to 
reach  a  training  criterion,  and  (3)  the  level  of  training  to  be 
conducted  on  each  training  device.  The  model  determines  the 
group  of  devices  that  should  be  used  to  train  each  task,  and  the 
level  of  training  at  which  a  student  should  change  from  one 
device  to  the  next  device  in  sequence,  in  order  to  achieve 
required  training  standards  at  minimum  total  training  cost. 

Model  outputs  include  the  associated  training  time  required  on 
each  device, 

0PT/A41I;  Determine  training  device  hourly  cost.  This 
activity  allocates  the  total  life  cycle  cost  of  a  training  device 
over  its  expected  number  of  hours  use.  The  estimate  is  based  on 
a  nominal  utilization  (UNOMj^)  in  hours  per  year.  In  the  initial 
run  of  the  model  CJNOMa  is  set  to  the  maximum  annual  utilization 
(UMAXj^)  found  in  the  data  base  (3A5)  or  calculated  in  0PT/A3234 
(03A3).  On  later  iterations,  the  value  of  UNOM^  that  is  updated 
in  activity  OPT/A4134  (0413A)  is  used  in  this  activity.  The 
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total  hourly  cost  for  the  device  (VTOTC^)  is  given  by  the 
following  equation: 

VTOTC^  =  INVi/(LCi  *  UNOM^)  +  FOCi/UNOM^  +  VHRC^, 

where  the  following  parameters  are  taken  from  the  data  base: 

INVj^  =  the  investment  cost  per  device  for  device  i  { 3A1 )  , 

LCi  =  the  number  of  years  in  the  life  cycle  for  device  i 
(3A4) , 

FOC^  =  the  annual  fixed  operating  cost  of  device  i  ( 3A2 ) ,  and 
VHRC^  «  the  hourly  variable  operating  cost  of  device  i  (3A3). 

0PT/A412:  Identify  cost  effective  devices.  This  activity 

analyzes  the  relationship  between  each  task  and  possible  training 
device  configurations  (selection  and  sequence)  to  determine  the 
training  device  configuration  that  can  train  from  entry  level  to 
training  standard  at  the  lowest  cost.  It  performs  this  analysis 
by  choosing,  for  each  performance  level,  that  device  providing 
the  highest  rate  of  increase  in  student  performance  as  a  function 
of  the  effective  price  of  using  that  device.  The  specific 
procedure  used  is  described  in  the  detailed  description  of  this 
activity. 
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OPT/A413:  Examine  Device  Utilization.  This  activity 

coiroares  the  value  of  device  utilization  that  was  assumed  in  the 
analysis  (UNOM^)  with  an  actual  utilization  value,  UACTj^, 
calculated  at  0PT/A4133.  If  UNOM^  and  UACT^  are  very  different 
for  device  i,  then  that  device  may  be  considerably  more  or  less 
efficient  than  was  assumed  by  the  analysis.  In  this  case  the 
user  may  decide  to  rerun  the  analysis  adjusting  UNOM^  to  have  the 
value  of  UACT^ .  The  specific  procedure  is  described  in  the 
detailed  diagram  for  this  node. 
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OPT/A412;  Identify  Cost  Effective  Devices 

This  activity  identifies  one  or  more  efficient  training 
devices  for  each  task.  The  basic  method  used  maximizes  the 
slope  of  the  curve  that  expresses  performance  on  a  task  as  a 
function  of  cost  for  each  training  device.  At  any  performance 
level,  the  method  chooses  the  training  device  with  the  highest 
slope,  and  continues  with  that  device  until  the  slope  for  that 
device  is  exceeded  by  another  device,  or  until  the  training 
standard  is  reached.  The  cost  effective  devices  are  determined 
using  the  following  five  steps. 

0PT/A4121:  Determine  first  device.  The  first  device  that 

is  chosen  maximizes  the  slope  of  the  learning  curve  with  respect 
to  cost  at  the  entry  performance  level.  The  slope  of  the 
learning  curve  with  respect  to  attributed  per-student  cost,  C, 
expressed  as  a  function  of  student  performance  level,  P,  is  given 
by  the  following  equation: 


dC 


ASM^^  *  r  •  TM^^  •  TSF^ 
VTOTC^ 


(  1 


P/ASM^. 


(r  +  l)/r 


where  VTOTC  is  the  hourly  cost  rate,  found  at  0PT/A411,  and 
where  the  following  learning  curve  parameters  used  (OID): 
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ASM,j.^  =  asymptote  of  learning  curve  for  task  T  and  device  i, 
r  =  learning  curve  exponent, 

TM-Pi  =  learning  curve  time  multiplier  for  task  T  and  device  i, 
TSF^  =  time  scaling  factor. 

Solving  this  equation  at  P  =  ENT^  gives  the  cost-effectiveness  of 
each  training  device  at  the  entry  level.  The  activity  then 
chooses  the  device  with  the  highest  slope.  The  currently  chosen 
device  will  be  termed  the  incumbent  device  and  will  be  denoted  by 
the  subscript,  s. 

0PT/A4122:  Set  current  performance  level.  The  current 

performance  level  keeps  track  of  the  effect  of  training  on 

performance.  Initially,  the  current  performance  level  is  set  to 
the  entry  performance  level  (ENT-p).  As  training  is  conducted  on 
a  training  device,  performance  improves.  When  a  change  is  made 
from  one  training  device  to  another  training  device  ( in 
OPT/A4124),  is  set  to  the  performance  level  at  the  slope 

crossover  point  (determined  in  OPT/A4123). 

0PT/A4123:  Find  device  crossover  points.  This  activity 

determines  the  performance  level  at  which  the  slope  of  the 
learning  curve  for  each  device  surpasses  that  of  the  curve  for 
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the  incumbent  device,  s.  The  crossover  point  (PXg^)  is  defined 
by  the  following  equation: 


PX  .  = 

SI 

where 


ASM,j,^  -  (ASM,j,g  ♦  z) 
1  -  z 


ASM 


z  = 


Ti 


ASM 


Ts 


ASM^g  TM,j,g  VTOTC^ 
ASM^  TM^  VTOTCg 


r/{r  +  1) 


0PT/A4124:  Process  crossover  points.  This  activity 

examines  the  crossover  points,  eliminates  training  devices  that 
are  no  longer  in  contention  for  this  task,  and  selects  the  best 
candidate  training  device  for  further  training.  The  activity 
uses  the  following  rules  to  process  the  crossover  points: 

1.  If  PXgj^  <  P^ow'  there  will  be  no  switch  to  training 

device  i  for  greater  performance  levels,  so  eliminate  it 
from  further  consideration. 
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2.  If  no  candidate  training  device  survives  the  previous  step, 
then  train  on  the  incumbent  device  to  the  training  standard. 
If  the  asymptote  of  the  incumbent  device  is  less  than  STD-p, 
then  the  performance  standard  cannot  be  met  using  the 
combination  of  devices  being  evaluated. 


3.  Otherwise,  choose  as  the  new  incumbent  device  the  device 
whose  crossover  point  is  reached  first,  i.e.,that  device 
with  the  smallest  PXg^.  Set  the  training  criterion  for 
device  s  to  this  minimum  PXg^.  This  value  will  also  become 
the  new  current  performance  level  when  it  is  processed  by 
OPT/A4122,  but  only  after  completion  of  0PT/A4125. 

OPT/A4125:  Calculate  training  time  and  cost.  This  activity 

determines  the  time  required  to  train  from  P^ow  next 

crossover  performance  level  (i.e.,  the  minimum  PXg^ )  on  the 
incumbent  device,  s.  The  training  time  (TRNT^^)  is  calculated 
from  the  Inverse  of  the  learning  function,  as  follows: 


TRNT 


Ts 


TM 


Ts 


TSF, 


(1 


-1/r 


-  (1 


-  Pnow/^'S'^Ts 


-1/r 


The  cost  is  calculated  by  multiplying  TRNT.j.g  by  the  total  hourly 
training  device  cost,  VTOTCg . 
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0PT/A413:  Examine  Device  Utilization 


In  this  activity  the  solutions  for  the  Individual  tasks  are 
aggregated,  and  an  overall  accounting  of  the  devices,  costs,  and 
training  time  is  produced.  In  addition,  assumptions  of  the  model 
regarding  training  simulator  utilization  are  checked,  and  the 
actual  utilization  is  presented  to  the  user,  so  that  the  user  may 
decide  whether  reanalysis  is  required. 

0PT/A4131:  Calculate  total  device  hours.  This  step 

accumulates  the  total  training  time  (TRNTy^)  over  tasks.  The 
resulting  device  training  time  (DEVT^)  is  simply  the  sum  of 
TRNTj^  over  all  tasks. 


0PT/A4132:  Calculate  required  number  of  training  devices. 

The  first  step  in  calculating  the  training  device  utilization 
(UACT|)  is  to  determine  the  number  of  training  devices  required 
to  implement  the  selection  (NDEVj^).  NDEVj^  is  calculated  from  t 
device  training  time  (DEVT^)  and  the  maximum  device  utilization 
(UMAXj^)  using  the  following  equation. 


NDEV^  =  ceiling 


DEVT^  *  GRADS 
UMAX, 
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where 

ceiling ( argument )  =  the  smallest  integer  at  least  as  great  as 

the  function  argument,  and  where 

GRADS  =  the  required  annual  student  throughput  ( 6A1 ) . 

0PT/A4133:  Determine  actual  utilization.  The  annual 

utilization  in  hours  (UACT^)  is  calculated  by  dividing  the  total 
number  of  hours  in  which  the  device  will  be  used  in  a  year  by  the 
number  of  simulators,  as  follows: 


UACT^ 


DEVT^  *  GRADS 

ndevT 


0PT/A4134:  Update  model  assumptions.  In  this  step,  the 

calculated  utilization  (UACT^)  is  compared  to  the  nominal 
utilizations  (UNOMj^)  assumed  in  the  model.  If  these  utilizations 
are  too  different  (e.g.,  2000  hours  versus  3000  hours),  the  user 
may  decide  to  revise  the  assumed  simulator  utilization.  In  this 
case  UNOMj^  will  be  assigned  the  value  of  UACT^.  When  the 
revisions  have  been  made,  the  analysis  will  be  repeated  to  obtain 
a  selection  based  on  the  revised  values. 
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0PT/A42;  Allocate  Resources  to  Training  Devices 

The  resource  allocation  activity  determines  a  refined 
training  program  minimizing  total  system  cost  to  train  students 
to  criterion  across  tasks,  subject  to  constraints  on  the  amount 
of  time  a  training  device  may  be  used  or  the  performance  level  at 
which  the  device  may  be  used.  It  employs  a  detailed  cost 
function  for  each  training  device. 

The  algorithm  to  solve  the  resource  allocation  problem  is 
heuristic  and  iterative.  The  general  idea  is  to  detail  the 
device  cost  curves  only  around  the  solution.  The  strategy  is  to 
begin  with  relatively  undetailed  cost  curves  and  generate  a  first 
solution  using  these  curves.  Then  the  cost  curves  are  detailed 
around  that  solution  and  the  cost  minimizing  solution  is  found 
anew.  If  the  new  solution  is  found  to  lie  within  the  cost  curve 
domains  detailed,  the  process  is  terminated,  and  the  solution  is 
deemed  optimal.  If  the  current  solution  does  not  lie  totally 
within  current  domains  of  detailing,  the  cost  curves  are 
redetailed  around  the  current  solution,  and  the  cost  minimization 
solution  found.  If  the  process  is  not  terminated  after  a 
predetermined  set  of  iterations,  the  process  is  terminated  and 
the  last  solution  found  deemed  "optimal",  even  though  it  lies 
outside  the  domain  of  detailing  for  one  or  more  devices.  In  any 
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case,  the  true  cost  functions  are  used  to  determine  the  per- 
student  cost  of  training  for  the  final  solution. 

OPT/A421:  Detail  Cost  Curves.  On  the  first  visit  to  this 

activity  each  cost  curve  is  detailed  into  (at  most)  a  two  segment 
continuous,  piecewise  linear  function.  The  first  segment 
corresponds  to  employment  of  training  device  assets  on  hand,  and 
the  final  segment  corresponds  to  acquisition  and  employment  of 
new  assets.  This  simple  two  segment  approximation  is  made  to  get 
the  iterative  solution  process  going. 

On  subsequent  visits  to  this  activity  the  cost  curves  are 
detailed  for  several  segments  on  and  around  the  per-student 
device  usage  found  in  the  preceding  solution.  This  will 
generally  be  a  discontinuous  piecewise  linear  curve — a  stairstep 
function  with  sloping  steps. 

On  each  visit  to  this  activity  the  number  of  segments 
detailed  must  be  limited  to  keep  the  total  number  of  combinations 
reasonable,  since  the  optimization  pioblem  will  be  solved  for 
most,  if  not  all,  of  the  cost  curve  segment  combinations.  If  the 
number  of  combinations  is  too  large,  then  some  of  the  curves  are 
further  simplified,  at  the  expense  of  accuracy.  The  curves  are 
selected  for  simplification  based  on  least  potential  error 
introduced  by  the  simplification. 
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OPT/A422:  Solve  Current  Multi-Task  Resource  Allocation 

Problem.  On  each  visit  to  this  activity  the  constrained  multi¬ 
task  resource  allocation  problem  is  solved  under  the  current  set 
of  device  cost  functions  detailed  in  0PT/A421.  The  solution 
procedure  involves  solving  the  problem  for  each  possible 
combination  of  cost  curve  segments. 

OPT/A423;  Check  Solution  Location  Vis-a-Vis  Detailing.  The 
solution  generated  in  0PT/A422  will  specify  a  certain  total  per- 
student  usage  for  each  training  device.  If  these  usages  fall 
within  the  domain  of  detailing  for  the  respective  device  cost 
functions,  then  the  solution  has  been  found  with  proper 
consideration  of  cost  curve  detailing.  If  not,  then  the  usage 
for  one  or  more  devices  falls  outside  the  domain  of  detailing,  so 
will  not  represent  desired  accuracy.  In  that  case,  another 
iteration  through  0PT/A421  and  OPT/A422  will  be  conducted,  time 
permitting,  in  an  effort  to  bring  the  solution  into  consonance 
with  the  domains  of  cost  curve  detailing. 

If  convergence  is  not  obtained  within  a  mandated  number  of 
iterations  through  0PT/A421  and  OPT/A422,  then  the  last  solution 
obtained  is  used,  and  the  concomitant  inaccuracy  in  cost  curve 
detailing  accepted.  Even  in  this  case  the  cost  of  the  final 
solution  is  evaluated  using  the  properly  detailed  device  cost 
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curves.  Thus,  failing  convergence,  the  solution  will  still  be 
feasible  and  accurately  costed,  but  is  expected  to  be  somewhat 
more  costly  than  a  truly  optimal  solution.  Experience  to  date 
suggests  that  the  optimization  error  will  tend  to  be  small  even 
if  convergence  is  not  achieved. 
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OPT/A421:  Detai,!  Cost  Curves 

This  activity  develops  a  detailed,  piecewise  linear  cost 
curve  for  each  training  device,  based  on  the  device  cost  data 
(3A).  Each  curve  incorporates  the  effects  of  discrete 
acquisition  of  devices,  but  not  economies  of  scale.  To 
facilitate  future  OSBATS  capabilities,  provision  is  also  made  to 
incorporate  device  assets  already  on  hand. 

In  principle,  each  cost  curve  consists  of  an  infinite  series 
of  steps,  but  the  optimization  algorithm  can  only  handle  cost 
curves  with  a  small  number  of  steps  at  any  one  time.  So  it  is 
necessary  to  detail  the  cost  curve  for  only  a  few  steps,  and  to 
represent  the  remainder  of  the  curve  by  (possibly)  one  finite 
linear  segment  preceding  the  domain  of  detailing  and  one  infinite 
linear  segment  following  it.  The  cost  curve  for  device  i  is 
formally  defined  by  the  following  parameters: 

t  =  Expected  total  time  that  an  individual  student 
will  use  the  training  device  in  question  during 
the  training  program. 

CUMCj(t)  =  Training  device  costs  attributed  to  each 

individual  student  if  each  student  uses  training 
device  for  amount  of  time  t. 

=  Number  of  segments  in  the  cost  curve. 


176 


Annual  Student  Throughput  &A1 


Title:  Pefferm  Prelifnmary  Pnooeasing 


I  Dale:  7/27/8#  Number 
I  ,  OPT-3 


First,  a  rule  base  is  interrogated  to  compare  task  data  and 
rule  conditions.  The  instructional  feature  rules  {5A)  associate 
instructional  features  with  specific  task  characteristics.  An 
example  of  such  a  rule  is  given  below: 

IF:  Entry  performance  ( lAl )  <  0.4, 

and  Intrinsic  feedback  (2B11)  is  absent, 
and  The  task  Involves  continuous  movement  (2B1), 
or  procedures  (2B2), 

or  decision  making/rule  using  (2B4), 

THEN:  Automated  Performance  Alerts  is  indicated  for  this  task. 

This  activity  compares  the  conditions  of  the  instructional 
feature  rules  to  the  task  characteristics  (2B)  and  task  learning 
points  (lA),  and  identifies  matches,  which  it  passes  on  to  the 
n^xt  activity. 

Next,  this  activity  takes  the  matches  produced  in  the 
interrogation,  and  sets  the  corresponding  cells  of  IFT,j.j^.  The 
IFT  matrix  is  defined  as  follows: 

TpT  _  f  1  if  ®  match  was  found  between  feature  k  and  task  T 
"Tk  ~  1  0  otherwise. 
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n  =  Running  index  on  total  number  of  devices  acquired 
(n=  0,1,2,  ...  ,  N-l) 

=  Starting  value  for  n-th  segment  of  cost  curve 
CUMCi { t) 

=  Height  of  cost  curve  CUMCj^(t)  at  beginning  of  n-th 
segment . 

=  Slope  of  n-th  segment  of  cost  curve  CUMCj^(t). 

With  two  exceptions,  each  segment  is  open  on  the  left  and 
closed  on  the  right.  The  first  exception  is  the  first  interval, 
which  is  a  single  point  t  =  0.  The  second  exception  is  the  final 
interval,  which  is  open  on  the  left  and  extends  with  constant 
slope  and  without  limit  on  the  right.  The  cost  curve  is  defined 
as  follows,  as  n  ranges  from  0  to 


CUMC^(t)  = 


A.  +  B.  *  (t 
in  in 

®iN; 


T.  ) 
in' 

T  ) 
IN,  ’ 


^  =  ^iO 

^in  <  ^  '■=  ^in.l 
^iN.  <  ^ 

I 


Discontinuities  occur  in  the  cost  curve  at  the  points  at 
which  the  training  requires  the  procurement  of  an  additional 


173 


Nom: 

TiOc'  Allocata  Training  Resources  to 

i  Date: 

I  Number  J 

OPT/A42 

1 

Training  Devices 

1  a/1/88 

j  OPT-31 

training  device.  These  discontinuities  occur  at  the  endpoints 
(Tin)  of  the  cost  curve. 

This  activity  restricts  the  numbers  of  segments  on  the 
various  curves  so  that  the  total  number  of  combinations  is  no 
more  than  MAXKB ,  the  maximum  number  of  combinations  allowed. 

This  is  accomplished  by  iteratively  decreasing  the  number  of 
segments  on  the  cost  curves.  The  process  begins  at  OPT/A4211  by 
detailing  the  cost  curves  within  an  initial  "window"  defined  in 
terms  of  a  domain  and  range  for  cost  curve  detail,  and  a  maximvun 
number  of  segments  allowed.  The  domain  is  selected  so  that  the 
curve  will  be  exact  in  the  region  of  current  interest,  i.e.,  at 
the  last  determined  value  of  device  usage.  If  the  total  number 
of  combinations  exceeds  MAXKB,  an  effort  is  made  to  reduce  the 
number  of  segments  on  the  various  curves  by  simply  shrinking  the 
"window"  within  which  each  curve  is  detailed  (OPT/A4212).  If 
this  fails  to  reduce  the  total  number  of  combinations 
sufficiently,  then  certain  curves  are  simplified  to  one  or  two 
linear  segments  (OPT/A4213);  the  curves  to  be  so  simplified  are 
chosen  to  minimize  the  maximum  error  potentially  introduced  by 
the  simplification. 

0PT/A4211:  Detail  Cost  Curves,  Within  Control  Window.  This 

activity  is  controlled  by  four  parameters:  KASE,  GRANGE,  TRANGE, 
and  NRANGE.  On  the  first  visit  to  this  activity  (KASE  =  1)  each 
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cost  curve  is  detailed  into  (at  most)  a  two  segment  continuous, 
piecewise  linear  function.  The  first  segment  corresponds  to 
employment  of  training  device  assets  on  hand,  and  the  final 
segment  corresponds  to  acquisition  and  employment  of  new  assets. 
This  simple  two  segment  approximation  is  made  to  get  the 
iterative  solution  process  going.  On  subsequent  visits  to  this 
activity  (KASE  =  2)  the  cost  curves  are  detailed  further  into 
segments  on  and  around  the  per-student  device  usage  found  in  the 
preceding  solution.  The  control  window  is  basically  defined  by 
GRANGE  and  TRANCE.  GRANGE  is  the  "height"  of  the  window,  i.e., 
the  range  within  which  the  curve  must  be  detailed;  it  is 
expressed  in  dollars  per  student.  TRANCE  is  the  "width"  of  the 
window,  i.e.,  the  domain  within  which  the  curve  must  be  detailed; 
it  is  expressed  in  hours  per  student.  The  parameter  NRANGE 
specifies  the  number  of  segments  desired,  and  may  be  considered 
another  control  window  parameter. 

On  the  first  pass  through  this  activity  three  important 
measures  of  regret  are  defined  for  each  training  device  i: 

RGRT21j^  =  the  increase  in  possible  cost  error  if  a  cost  curve  of 
a  single  infinite  segment  is  used  instead  of  a  curve  with  two 
segments;  RGRT31j^  =  the  increase  in  possible  cost  error  if  a  cost 
curve  of  a  single  infinite  segment  is  used  instead  of  a  curve 
with  three  or  more  segments;  and  RGRT32^  =  the  increase  in 
possible  cost  error  if  a  cost  curve  of  a  two  segments  is  used 
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instead  of  a  curve  with  three  or  more  segments.  These  regrets 
are  determined  by  first  calculating  ERRl,  the  maximum  error  that 
can  occur  if  the  curve  is  simplified  to  a  single  infinite 
segment,  and  ERR2 ,  the  maximum  error  that  can  occur  if  the  curve 
is  simplified  to  exactly  two  segments.  If  the  curve  has  three  or 
more  segments,  there  will  be  no  error  for  student  usage  times 
falling  in  the  middle  segments.  If  there  are  no  assets  on  hand, 
then  ERRl  equals  the  step  height, 

STEPHT^  =  (INV^/LC^  +  FOCi ) /GRADS  . 

Since  curves  of  2  segments  offer  no  advantage  over  curves  of  1 
segment  when  there  are  no  assets  on  hand,  ERR2  is  set  to 
infinity.  If  there  are  assets  on  hand,  then  ERRl  is  computed 
based  on  a  tentative  two-segment  detailing: 

ERRl  =  maximum  (STEPHT^  -  ( Bi2”®il ^ *^i2 '  ^®i2~®il ^ *^i2) • 

In  the  case  of  assets  on  hand,  ERR2  =  STEPHTj^ . 

Once  the  error  terms  determined,  then  the  (potential)  regret 
in  going  from  a  curve  with  2  segments  to  a  curve  with  1  segment 
is  RGRT21^  =  ERRl  -  ERR2 .  The  regret  in  going  from  a  curve  with 
3  or  more  segments  to  one  with  2  segments  is  RGRT32^  *  ERR2  since 
there  is  no  error  if  curves  of  3  or  more  segments  are  used  and 
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Student  usage  falls  in  the  middle  segment(s),  a  condition  that 
will  be  forced  by  the  optimization  algorithm.  Similarly,  RGRTSl^ 
=  ERRl. 

OPT/A4212:  Simplify  Using  Control  Window  Shrinkage.  The 

number  of  combinations  of  device  cost  curve  segments  is 
determined  as  NCOMBO  =  NTAB^  *  NTAB2  *  ...  *  NTABjjq  where  NTAB^ 
is  the  number  of  segments  in  the  cost  curve  for  device  i,  and 
where  ND  is  the  number  of  training  devices  under  consideration. 

If  NCOMBO  exceeds  MAXKB,  then  the  size  of  the  control  window  is 
reduced  by  25a^  and  0PT/A4212  is  revisited  to  detail  the  cost 
curves  again  within  the  smaller  window.  More  precisely,  the 
following  substitutions  are  made:  GRANGE  =  0.75  *  ORANGE,  TRANCE 
=  0.75  *  TRANGE,  and  NRANGE  =  0.75  *  NRANGE  (integer  truncation). 
If  NRANGE  is  reduced  to  zero,  then  this  activity  is  terminated 
and  control  is  transferred  to  0PT/A4213  to  continue  the 
simplification  process. 

0PT/A4213:  Simplify  Using  Detail-Sensitivity  Ordering. 

This  activity  is  invoked  if  0PT/A4212  fails  to  reduce  the  number 
of  combinations  of  device  cost  curve  segments  to  an  acceptable 
level.  It  is  then  necessary  to  simplify  one  or  more  of  the 
device  cost  curves  to  only  one  or  two  segments.  Such 
simplification  will  generally  cause  a  loss  of  accuracy  in  the 
cost  computation.  In  order  to  minimize  the  impact  of  such 
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simplification,  the  curves  to  be  simplified  are  chosen,  in  turn, 
to  introduce  minimum  regret,  based  on  the  measures  RGRT21^, 
RGRTSlji,  and  RGRT32^  computed  in  OPT/A4211. 

Once  the  curve  to  be  simplified  next  is  identified,  together 
with  the  desired  number  of  segments,  the  simplification  is 
accomplished  by  repeating  (if  necessary)  the  KASE  =  1 
computations  of  0PT/A4211.  This  yields  a  one  or  two  segment 
curve.  If  a  two  segment  curve  is  constructed,  but  a  one  segment 
curve  is  desired,  the  curve  is  simplified  to  one  segment  by 
substituting  8^2  for 
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OPT/A4211;  Detail  Cost  Curves  Within  Control  Window 

This  activity  begins  by  constructing  a  one-  or  two-segment 
curve  accounting  for  assets  on  hand,  if  any.  Then,  if  further 
detailing  is  desired  (KASE=2),  a  segment  is  added  on  top  of  the 
current  per-student  usage  time.  Additional  segments  are  added  to 
the  left  and  right  of  this  segment,  until  the  desired  number  of 
segments  (NRANGE)  is  achieved  or  the  detailing  exceeds  one  of  the 
control  window  dimensions  (TRANCE,  GRANGE).  The  control  window 
parameters  are  treated  by  this  activity  as  guidelines  more  than 
strict  upper  bounds.  They  are  not  used  to  impose  a  Procrustean 
simplification  to  the  cost  curve. 

The  k-th  cost  curve  segment  for  device  i  is  defined  by  a 
starting  time  a  height  Aj^j^,  and  a  slope  There  is  no 

need  to  define  the  end  of  the  segment  since  it  simply  occurs  when 
the  next  segment  begins . 

0PT/A42111:  Construct  Base  Two-Seament  Cost  Curve.  This 

activity  begins  by  constructing  the  first  segment,  anchored  at 
the  origin,  so  that  T^i  =  0  and  A^j^  =  0.  If  there  are  assets  on 
hand,  then  the  slope  or  the  first  segment  is  =  FOCj^/UMAX^  + 

VHRC^,  which  assumes  a  pro  rata  per-student  share  of  fixed 
operating  costs,  together  with  variable  operating  costs,  but 
excludes  investment  costs,  since  the  assets  are  already  owned. 
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If  there  are  no  assets  on  hand,  then  this  first  summary  segment 
includes  investment  costs  and  is  calculated  as 

=  (INV^C/LCi  +  FOCi)/UMAXi  +  VHRCi. 

A  second  segment  is  constructed  only  if  there  are  assets  on 
hand.  In  that  case,  the  beginning  of  the  second  segment  is  found 
as  Tj^2  =  ASSETSj^*UMAXj /GRADS ,  i.e.,  the  utilization  at  which 
assets  on  hand  are  fully  used.  The  beginning  of  the  second 
segment  is  set  to  A^2  =  -  Tj),  i.e.,  the  height  of 

the  first  segment  at  its  end.  The  slope  of  the  second  segment  is 
then  set  to  3^2  =  (INV^/LC^  +  FOCi)/UMAXi  +  VARC^. 

OPT/A42112:  Add  Next  Segment  At  Current  Usage .  If 

additional  detailing  is  desired  (KASE=2),  then  that  segment  for 
the  ideal  curve  that  is  located  atop  the  current  device  usage 
TTOTj^  is  next  spliced  into  the  base  curve.  To  do  this  it  is 
necessary  to  determine  the  number  of  new  devices  associated  with 
TTOT^:  NEW  =  TTOT i* GRADS/ UMAX ^  -  ASSETS^,  rounded  up  to  the 

nearest  integer.  If  NEW  =  0,  then  no  new  devices  need  be 
acquired  to  satisfy  TTOTj^,  so  the  base  curve  is  adequate  as  it 
stands  and  no  detailing  is  required.  However,  the  domain  and 
range  of  detailing  are  marked  by  the  endpoints  of  the  first 
segment,  (TLEFT ,  CLEFT)  =  ( TRIGHT ,  CRIGHT )  = 

(Ti2,  Aj^2‘’‘®ii  *  ( )  .  Also,  two  parameters,  NSEGL  and 
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NSEGR,  marking  the  leftmost  and  rightmost  deliberately  detailed 
segment,  are  initialized  equal  to  1.  Two  other  parameters,  NEWL 
and  NEWR,  indicating  the  number  of  new  devices  associated  with 
these  two  segments,  are  initialized  equal  to  0. 


If  NEW  >  0  then  a  segment,  tentatively  numbered  will 

be  added.  (N^  is  either  1  or  2 ,  depending  on  whether  ASSETS  =  0 
or  ASSETS^  >  0,  respectively.)  This  is  done  by  redefining  = 
Nj+l  and  setting 

^iN  “  (ASSETS^  +  NEW  -1 ) »UMAX^/ GRADS 


^iN.  ~  ^iN.-l  ®iN,-l  *  ^"^iN.  ^iN.-l^ 

1  1  i  a  1 


+  (INV^/LC^+  FOC^) /GRADS 


BiN  =  VHRC^  . 
1 


The  domain  and  range  of  detailing  is  determined  as  the  segment 
endpoints:  (TLEFT,  CLEFT)  =  (T.„  ,  A.„  )  and 

i  1 

(TRIGHT,  CRIGHT)  =  ( TLEFT+UMAX . /GRADS ,  A.„  +  *  UMAX . /GRADS ) . 

I  IN .  IN .  i 

1  1 
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If  NEW  >  1,  then  the  segment  Nj^-1  that  was  previously  the 
last  segment  needs  to  be  retained.  But  if  NEW  =  1  and  ASSETS^  > 
0,  this  segment  is  dispensable,  since  the  new  segment  N^  begins 
before  or  at  the  same  point  as  that  segment;  in  that  case 
segment  N^-i  is  replaced  in  toto  by  segment  Nj^,  and  N^  is 
redefined  as  Nj^-1.  However,  if  NEW  =  1  and  ASSETS =  0,  the 
segment  N^-1  (which  is  anchored  at  ^he  origin)  is  retained  so 
that  the  cost  curve  will  give  a  zero  cost  in  response  to  a  zero 
per -student  time,  rather  than  the  value  STEPHT^^  >  0. 

Upon  completion  of  the  detailing  of  the  segment  N^  at  the 
current  usage,  NSEGL  and  NSEGR  are  initialized  equal  to  Nj^ ,  and 
NEWL  and  NEWR  are  Initialized  equal  to  NEW.  Then  a  new  final 
infinite  segment  is  added  by  setting 


=  TRIGHT 
=  CRIGHT 

=  (INV./LC.  +  F0C,)/UMAX.  +  VHRC,  . 
I  I  i  1  i 


^1  = 

^iN, 


iN, 


®iN, 
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OPT/A42113:  Add  New  Left  Segment.  At  this  point  a  check  is 

made  to  see  if  further  detailing  is  required,  based  on  the  number 
of  segments,  the  time  domain  covered,  or  the  cost  range  covered. 
In  particular,  if  NSEGL  >  1  and  NSEGR  -  NSEGL  +  3  <  NRANGE,  or  if 
NSEGL  =  1  and  NSEGL  -  NSEGR  +  2  <  NRANGE,  then  the  number  of 
detailed  segments  is  deemed  deficient.  If  TRIGHT  -  TLEFT  < 
TRANGE,  then  the  time  domain  detailed  is  deemed  deficient.  If 
CRIGHT  -  CLEFT  <  GRANGE,  then  the  cost  range  detailed  is  deemed 
deficient.  If  these  tests  indicate  that  further  detailing  is 
required,  a  new  segment  is  detailed  to  the  left  of  the  current 
detailed  domain.  However,  if  detailing  a  new  right  segment  would 
cause  TTOTj^  to  fall  more  near  to  the  center  of  [TLEFT,  TRIGHT], 
then  this  step  is  skipped  at  this  time. 

This  detailing  is  begun  by  setting  NEW  =  NEWL-1;  NEW  will 
be  the  number  of  new  devices  corresponding  to  the  segment  being 
detailed.  If  NEW  =  0,  then  no  more  detailing  is  possible  to  the 
left,  so  nothing  is  done  to  the  curve.  However,  the  control 
parameters  corresponding  to  the  leftmost  detailed  segment  are 
updated,  provided  it  is  not  the  first  segment  of  the  curve.  In 
particular.  If  NSEGL  >  1  then  NSEGL  is  set  equal  to  NSEGL  -  1, 
TLEFT  is  set  equal  to  (for  the  redefined  NSEGL),  and 

CLEFT  is  set  equal  to 


If  NEW  >  0,  then  there  is  room  for  another  segment 
corresponding  to  new  device  acquisitions  on  the  left.  The 
parameter  NEWL  is  set  equal  to  NEW.  A  new  segment  will  be  added 
at  the  old  left  segment  index,  NSEGL,  unless  NEW  =  l  and  NSEGL  = 
3,  in  which  case  the  old  segment  number  2  at  the  beginning  of  the 
post-fixed-assets  portion  of  the  base  curve  is  replaced.  In  that 
special  case  of  NEW  =  1  and  NSEGL  =  3,  the  left  segment  index  is 
shifted  to  NSEGL  =  NSEGL  —  1  to  force  the  old  segment  to  be 
replaced  by  the  operations  described  in  the  next  paragraph. 

If  NEW  >  0  and  either  NEW  is  unequal  to  1  or  NSEGL  is 
unequal  to  3,  then  the  segments  to  the  right  of  NSEGL  are 
renumbered  by,  essentially,  adding  l  to  the  corresponding 
indices.  Also  N^  is  set  to  N^  +  1  and  NSEGR  is  set  to  NSEGR  +  1. 
The  segment  NSEGL  is  then  redefined  by 

^iNSEGL  (ASSETS^  •*-  NEWL  -  1)  *  UMAX^/GRADS) 

^iNSEGL  "  ^i{NSEGL-l) 

■^®i  (NSEGL-1  )  *  ^"^iNSEGL  "  ^i(NSEGL-l)^ 

+  (INV^/LC^  +  FOC^) /GRADS 

®iNSEGL  =  V«^Ci  • 
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TLEFT  and  CLEFT  are  set  to  and  A^^^gg^g,  respectively. 

0PT/A42114:  Add  New  Right  Segment.  At  this  point  a  check 

is  made  to  see  if  further  detailing  is  required,  using  the  same 
logic  as  at  the  beginning  of  OPT/A42113.  The  actual  segment 
addition  is  more  straightforward  on  the  right,  since  there  are 
none  of  the  complications  associated  with  the  origin  or 
transition  from  assets  on  hand  to  new  acquisitions.  The  actual 
addition  of  the  segment  is  accomplished  by  setting  NEW  =  NEWR+1, 
NEWR  =  NEW,  NSEGR  =  NSEGR+1 , 


’’“^iNSEGR 

^iNSEGR 

®iNSEGR 

TRIGHT 

CRIGHT 


(ASSETS^  +  NEWR  -  1)  *  UMAX^ /GRADS) 
*1 ( NSEGR- 1 ) 

■*'®i  ( NSEGR- 1)  *  ^^iNSEGR  "  "^KNSEGR- 
+  (INV^/LC^  +  FOC^) /GRADS  . 

^  VHRC^  . 


"^iNSEGR 

^iNSEGR 


+  UMAX ^/ GRADS 


®iNSEGR  *  «MAX./GRADS  . 


1) 
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Then  a  new  final  segment  is  added  by  setting 


TRIGHT 
CRIGHT 

{INV^/LC^  +  FOC^)/UMAX^  +  VHRC^ 

OPT/A42115:  Check  for  Completion  and  Terminate.  At  this 

point  another  check  is  made  to  see  if  further  detailing  is 
required,  using  the  same  logic  as  at  the  beginning  of  OPT/A42113. 
If  further  detailing  is  required,  control  is  transferred  to 
0PT/A42113  to  begin  another  cycle  of  left-right  segment 
detailing. 

If  sufficient  detailing  is  accomplished,  a  check  is  made  to 
see  if  there  are  segments  lying  to  the  left  of  the  detailed 
domain,  l.e.,  it  NSEGL  >  1.  If  so,  those  segments  are  removed  in 
order  to  reduce  the  number  of  segments  in  the  curve. 


iN, 


‘IN, 
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1  1 

OPT/A422:  Solve  Multi-Task  Resource  Allocation  Problem. 

The  minimum  cost  resource  allocation  problem  is  solved  in 
this  activity,  given  the  currently  detailed  cost  curves.  The 
discontinuous  cost  curves  obviate  an  elegant,  one-shot  solution 
procedure.  Instead,  the  minimum  cost  resource  allocation  problem 
is  solved,  in  principle,  for  each  possible  combination  of  cost- 
curve  segments.  The  solution  generated  for  a  given  combination 
is  then  evaluated  using  the  true  cost  curves.  That  solution 
yielding  the  least  total  cost  is  deemed  the  optimal  solution  for 
the  current  set  of  cost  curves. 

Although  the  minimum  cost  resource  allocation  problem  is,  in 
principle,  solved  for  each  combination  of  cost  curve  segments,  it 
is  sometimes  possible  to  determine  that  a  given  combination  will 
not  possibly  yield  a  superior  solution.  This  is  accomplished  by 
examining  the  least  potential  cost  associated  with  the  given  cost 
curve  combination.  If  that  cost  exceeds  the  minimum  cost 
solution  found  up  to  that  point,  then  the  associated  cost  curve 
combination  is  ruled  out  of  consideration.  Further,  cost  curve 
combinations  are  examined  in  order  of  increasing  minimum 
potential  costs.  If  any  combination  is  deemed  excessively  costly 
compared  to  the  incumbent  solution,  then  all  subsequent  cost 
curve  combinations  are  then  known  to  be  excessively  costly,  too, 
so  processing  for  the  current  cost  curve  detailing  is  terminated. 
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{This  is  an  application  of  the  branch-and-bound  technique  of 
operations  research.)  The  required  ranking  of  cost  curves  by 
minimum  potential  cost  is  accomplished  at  OPT/A4221. 

For  a  given  combination  of  cost  curve  segments,  the  multiple 
task  constrained  resource  allocation  problem  is  solved  by 
iterating  between  nodes  0PT/A4222  and  0PT/A4223.  A  tentative 
solution  is  generated  at  OPT/A4222,  assuming  no  constraints  on 
device  usage.  The  mandated  device  usage  is  then  examined  at 
OPT/A4223,  in  light  of  actual  device  usage  constraints.  If 
davice  usage  constraints  are  violated,  then  artificial  surcharges 
are  applied  to  the  slopes  of  the  current  cost  curve  segments, 
making  the  overused  devices  appear  more  expensive,  and  the 
problem  is  solved  again  at  0PT/A4222.  After  each  iteration  it  is 
determined  whether  the  solution  respects  prescribed  usage 
constraints  and  the  surcharges  satisfy  associated  optimality 
conditions.  If  so,  the  solution  is  deemed  viable  and  optimal  for 
the  current  cost  curve  segments.  Otherwise,  iteration  continues. 
If  no  satisfactory  solution  is  found  after  a  prescribed  set  of 
iterations,  or  if  a  suspicious  "flip-flopping"  behavior  is 
observed  in  the  surcharges,  then  the  particular  cost  curve 
segment  combination  is  deemed  infeasible  and  is  abandoned.  (This 
use  of  surcharges  to  force  a  solution  within  prescribed 
constraints  is  an  application  of  the  sequential  unconstrained 
minimization  technique  (SUMT)  of  operations  research). 
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1  Node: 

1  Title:  Allocate  Training  Resources  to 

1  i 

OPT/A42 

1  Training  Devices 
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B/1/88  1 

NumDer 

OPT-31 


OPT/A4221:  Rank  Cost  Curve  Combinations  by  Minimum 

Potential  Costs.  The  lower  bound  CLBj^Qj^gQ  on  possible  per- 
student  cost  is  found  for  each  combination,  indexed  KOMBO,  of 
cost  curve  segments  by  summing  the  minimum  cost  for  the  active 
segment  for  each  device  cost  curve; 

^^®KOMBO  ^  ^ik^ 

where  kj^  is  the  cost  curve  segment  being  used  for  device  i  under 
combination  KOMBO.  A  ranking  array  KBRANK{KB)  is  then  determined 
by  sorting  the  combinations  by  increasing  cost.  Then  when  KB=1 , 
KBRANK(KB)  is  the  number  of  the  combination  having  the  minimum 
value  of  least  potential  cost,  based  simply  on  the  summed  lower 
edges.  Similarly,  KBRANK(NKOMBO)  would  be  the  number  of  the 
combination  having  the  greatest  value  of  least  potential  cost. 

It  is  important  to  understand  that  "least  potential  cost"  refers 
simply  to  the  least  cost  that  could  be  associated  with  the  given 
segments,  and  that  there  is  yet  no  implication  on  the  feasibility 
of  training  effectively  at  the  associated  device  usages. 

OPT/A4222:  Determine  Multi-Task  Resource  Allocation  for 

Specific  Cost  Curve  Combination.  This  activity  involves  first 
setting  up  a  set  of  single  task  problems,  using  current 
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performance  and  cost  data,  user-imposed  constraints,  and  cost 
curve  segment  surcharges  (0PT/A42221 ) ,  and  then  solving  these 
single  task  problems,  in  sequence  { OPT/A42222 ) .  For  each  device 
i  the  total  allocation  TTOTj^  is  found  as  the  sum  of  the 
allocations  TRNT^.p  over  the  tasks,  assuming  each  single-task 
problem  is  soluble. 

OPT/A4223:  Characterize  Solution  and  Adjust  Surcharges  to 

Force  Constraint  Compliance.  At  this  activity  the  complete 
multiple  task  solution  is  checked  for  constraint  violations  and 
surcharges  are  adjusted  correspondingly  (0PT/A42231 ) .  Then  a 
check  is  made  to  see  if  the  solution  and  surcharges  satisfy 
conditions  allowing  declaration  of  a  solution  for  the  multiple 
task  resource  allocation  problem,  or  whether  further  iteration  or 
abandonment  are  in  order  (0PT/A42232) . 
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Current  Resource  0422A 
Allocation  Solution 


Nod«: 

Title:  Solve  Mulb-Task  Resource  Allocation 

Date: 

Number 

OPT/A422 

KTOoisni 

af1/8B 

OPT-34 

OPT/A4222:  Determine  Multi-Task  Resource  Allocation  for  Specific 

Cost  Curve  Combination. 

OPT/A42221:  Conduct  Checks  and  Initializations  for  Current 

Combination.  The  minimum  potential  cost,  CLBj^Qj^gQ,  associated 
with  combination  KOMBO  =  KBRANK(KB)  is  compared  to  CBEST,  the 
per-student  cost  of  training  under  the  incumbent  best  solution 
(CBEST  is  initialized  as  infinity).  If  CLBj^Qj^gQ  exceeds  CBEST, 
then  KOMBO  can  not  possibly  yield  a  better  solution,  nor  can  any 
of  the  following  combinations,  associated  with  KB+1 ,  KB+2,  etc.; 
so  in  that  case  0PT/A422  processing  is  terminated. 

Otherwise,  combination  KOMBO  is  processed.  For  each 
training  device,  i,  the  maximum  allowed  per-student  device  usage 
is  set  equal  to  the  minimum  of  the  user-imposed  usage,  USMAXj^, 
and  the  usage  corresponding  to  the  end  of  the  assumed  cost  curve 
segment,  LEVj^,  for  the  device.  I.e., 

TAVAIL,  =  Minimum  (USMAX.,  T.,^„„  )  , 


where  the  end  of  the  LEV^^  segment  is  determined  by  the  beginning 
of  the  LEV^+1  segment.  This  constraint  is  applied  as  an  upper 
bound  to  device  i  allocations  for  each  task  T,  also:  UB^^  = 
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TAVAIL^.  (This  constraint  will  not  prevent  overallocation  to 
device  i,  since  allocations  for  several  tasks  may  add  to  a 
violation,  but  it  prevents  overallocation  for  a  single  task.) 

Also  in  this  activity  the  cost  surcharge  SURCHG^  for  each 
device  i  is  initialized  at  1 . 

OPT/A42222:  Set  Up  and  Solve  Single-Task  Subproblems.  On 

each  visit  to  this  activity  an  effective  price  for  each  device  i 
is  set  by  applying  the  current  surcharge  SURCHG^: 

=  SOROHGj  .  . 

i 

On  the  first  visit  to  this  activity  each  surcharge  has  the 
initialized  value  SURCHG^  =  1,  corresponding  to  no  penalty  for 
usage.  On  subsequent  visits  to  this  activity,  as  iteration  is 
accomplished  to  force  overall  device  usage  constraint 
accommodation,  the  adjusted  surcharges  are  used,  as  set  in 
0PT/A42231 . 


At  this  point  a  simple  single-task  constrained  resource 
allocation  problem  is  defined  for  each  task; 
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Find  a  sequence  of  devices,  IDV^ ,  IDV. 


and  associated  per-student  training  allocations  TRNT 


TRNT  . TRNT 

k'  •^^''nstage^'^ 

T 


NSTAGE„ 


IDVj,T . 


that  will  minimize 

[•AGE^ '  ^ 

T 

Total  Cost  =  I  TRNT^^ 


subject  to 

I  TRNT.^  < 

Student  Performance  >  EPMIN. 
on  task  T  prior  to 
training  on  device  i 


where  student  entry  performance  to  the  task  is  PENTRY^  and  where 
student  performance  at  the  end  of  training  on  each  device  is 
determined  by  the  learning  curve  for  that  device  on  that  task. 

The  actual  single-task  optimization  algorithm  is  based  on 
mathematical  analysis  to  determine  optimal  switching  between 
devices  in  an  unconstrained  environment,  and  iterative  techniques 
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to  lock  in  allocations  and  proceed  to  a  simplified  problem  in 
order  to  handle  the  allocation  and  performance  constraints.  The 
algorithm  is  detailed  in  OPT/A422221  -  OPT/A422225, 
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OPT/A42222:  Set  Up  and  Solve  Single-Task  Subproblems 

In  order  to  simplify  this  discussion  it  is  convenient  to 
recapitulate  the  precise  problem  definition,  but  suppressing  any 
reference  to  the  task  T,  which  is  an  implicit  index  on  all 
variables,  as  indicated  in  the  OPT/A4222  discussion  of  this 
activity.  Let 

i,  j  =  Indexes  referring  to  the  training  devices 
TRNT^  =  allocation  of  per-student  time  to  device  i 

=  Attributed  per-student  variable  cost  rate  for 
training  on  device 

p.  =  performance  at  completion  of  k-th  stage 
•^k  training,  on  device 
UBj  =  user  imposed  upper  bound  on  per-student 
•training  time  for  device  j. 

PERF^  ( TRNT^  ,  PyjQj^)  =  performance  achieved  on  training  device  i 

after  training  an  amount  of  time  TRNT^ ,  given 
an  entry  level  of  performance 
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The  function  PERF^  { t ,  is  evaluated  using  the  associated 

inverse  function  PINV^(p)  as  follows: 


PINVi(p)  = 


(1  -  p/ASM^)  1 

TM^  *  TSF 


The  first  step  in  evaluating  PERF^  ( t ,  Pjjq„)  is  then  to  determine 
the  "head  start"  tize  parameter  associated  with  the  entry 
performance  Pnow=  =  PINV^  ( PENTRY)  .  Then  PERF^  ( t ,  P^o;^) 

is  calculated  as 


PERFjit,  P„^„)  =  ASM 


J  [7  -  [T  +  TM^ 


*  TSF  *  ( 


t  +  HSX]  . 


With  these  definitions  in  hand,  the  single-task  optimization 
problem  becomes: 

Determine  a  sequencf*  NSTAGE  training  devices 

and  associated  per-student  training  allocations  {TRNT^^j^  )  to 
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and  subject  to  performance  constraints  and  relations 


PERF, 


(TRNT. 


NSTAGE 


NSTAGE 


'  ^IDV 


)  =  Pc 


NSTAGE-1 


where  p .  and  the  other  device  j.  completion  performance  levels 

p.  are  defined  recursively  by 
^k 

p  =  PERF  (TRNT  .  p  ) 

•'k  -^k  ■'k  ■'k-l 

and  where  the  entry  performance  to  device  j^  is  set  at  the 
student  entry  performance  level: 

Pj„  ^  ^ENTRY 


The  algorithm  may  be  considered  a  superset  of  the  training 
device  selection  procedures  defined  in  0PT/A41,  and  shares  many 
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of  the  underlying  equations.  In  fact,  an  identical  solution  will 
be  obtained  for  an  unconstrained  problem.  However,  the  present 
algorithm  involves  more  complex  heuristic  procedures  to  handle 
constraints.  It  contrasts  with  0PT/A41  by  allocating  "from  the 
top  down",  i.e.,  by  first  determining  that  training  device  to  be 
used  on  the  final  stage  of  student  training,  then  determining 
that  device  to  be  used  on  the  penultimate  stage,  and  so  on. 

(This  "backwards"  allocation  was  found  preferable  for  handling 
per f ormance-and  resource-constraints.)  More  than  one  pass  may  be 
required  in  order  to  avoid  using  the  same  device  at  two  distinct 
stages  of  training,  an  eventuality  that  will  not  be  allowed.  A 
tentative  selection  of  a  device  at  mor£  than  one  point  may  occur 
if  it  must  be  used  in  an  inefficient  manner,  in  terms  of  learning 
per  dollar,  because  a  more  efficient  device  is  proscribed  by 
constraints.  In  such  a  case,  the  training  resources  assigned  at 
different  stages  to  the  same  device  on  one  pass  are  "lumped" 
together  on  the  next  pass  in  a  forced  allocation  at  that  po-int 
where  the  device  in  question  is  first  selected  for  employment. 

OPT/A422221:  Begin  Next  Algorithm  Pass.  Processing  begins 

by  initializing  all  allocations  to  zero,  and  setting  the  current 
performance  level  to  the  standard,  P^ow  “  preparation 

for  the  "backwards"  allocation  procedure  The  pass  counter, 

PASS,  is  incremented,  and  the  index  of  the  incumbent  device, 
INCMB,  is  set  to  a  value  flagging  it  as  unassigned. 
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i  I 

OPT/A422222;  Determine  Device  for  Next  Step  in  Backward 
Allocation.  The  device  to  be  used  for  training  up  to  the  current 
performance  level,  P^qw'  selected  as  that  qualified  device 
providing  the  greatest  rate  of  increase  in  performance  with 
respect  to  per-student  dollar  spent  at  that  performance  level. 

For  device  i  this  rate  is  given  by 


PRATE ^ 


r  +  1 


ASM^  •  r  *  TM^  *  TSF 

r  ‘’now  n 

L 

I  _ 

I 

where  ASM^ ,  r,  TM^ ,  TSF  are  the  asymptote,  shape  parameter,  time 
multiplier,  and  time  scaling  factor  for  the  learning  curve  for 
device  i  (for  task  T)  ,  and  where  is  the  effective  per-student 
price  for  usage  of  device  i.  A  device  is  deemed  qualified  for 
consideration  if  it  is  not  the  incumbent  device,  if  it  can 
actually  be  used  to  train  to  level  (i.e.,  ASMj^  >  Pnow^  ' 

there  are  appreciable  training  resources  left  (i.e.,  TNRT^  > 

UBj),  and  if  the  current  performance  level  exceeds  the  minimum 
allowable  entry  performance  level  for  the  task  in  question  (i.e., 
p  >  EPMINi).  If  no  qualified  devices  are  available,  the 
problem  is  deemed  Insolvable,  and  OPT/A42222  is  abandoned,  with  a 
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OPT-36  ( 

i  1 

flag  CONDTN  set  to  Indicate  that  no  solution  was  found.  If 
qualified  devices  are  available,  that  one  with  the  greatest 
PRATEj^  is  labeled  is  selected  as  the  new  incumbent,  I. 

OPT/A422223:  Determine  Next  Switching  Performance  Level . 

The  next  (actually,  predecessor)  device  to  be  employed  in  the 
training  sequence  is  now  determined  based  on  ( 1 )  ideal 
performance  levels  for  economic  transfer  to  other  devices,  (2) 
available  resources  for  incumbent  device,  and  (3)  minimum  entry 
performance  constraints. 

During  this  activity  the  next  device,  INEXT,  and  the 
transition  performance  level,  P^ext'  determined.  Pnext 

initialized  at  zero,  but  immediately  increased  to  PgNTRY' 
is  greater,  and  increased  further  to  EPMINj  if  this  is  greater. 
The  minimum  performance  level,  Pmini '  that  can  be  reached  using 
the  incumbent  device  (going  "backwards"  down  the  learning  curve) 
from  P^iow  determined  based  on  the  remaining  unallocated 
resources,  UBj  -  TRNTj : 

Pmini  =  perFi(-[uBi  -  trntj],  p^^w)  • 

If  this  is  greater  than  P^ext'  then  P^ext  increased  to  P^inI ' 
since  the  incumbent  device  can  reach  no  lower  than  P^ini • 
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The  performance  level  PXjj  at  which  the  incumbent  device  I 
has  the  same  learning  rate  per  dollar  as  a  challenger  device  J  is 
computed  in  two  steps,  involving  an  intermediate  variable  z: 


PXjj  =  (1  -  z) 


ASMj  *  ASMj 

ASMj  -  ASMj  *  z 


where 


ASMj  •  TMj  * 

ASMj  *  TMj  *  Bj 


(Since  PXjj  will  be  the  same  at  any  point  in  the  single-task 
resource  allocation  algorithm,  it  is  only  computed  once,  and  a 
flag  set  to  avoid  later  recomputation.  Also,  when  PXjj  is 
computed,  PXjj  is  then  set  equal  to  PXjj  since  they  are  equal; 
this  obviates  later  calculation  of  PXjj . ) 


If  PXjj  exceeds  both  P^ext  EPMINj,  and  is  less  than 

Pnow'  then  the  economic  transition  to  J  will  be  encountered 
before  that  to  any  other  device  considered  prior  to  J. 
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Therefore,  J  becomes  the  tentative  next  device  INEXT  =  J,  and 
Pnext  equal  to  PXjj. 

0PT/A422224:  Allocate  Resources  to  Step,  Forcing  Lumping 

from  Previous  Pass.  At  this  point  resources  are  actually 
allocated  to  the  selected  incumbent  device,  I.  A  counter  KALLOCj 
is  incremented  to  keep  track  of  the  number  of  times  device  I  has 
received  allocations  on  this  pass.  If  KALLOCj  becomes  greater 
than  one,  then  another  indicator,  ALLOCM,  is  set  from  "no"  to 
"yes"  to  indicate  that  multiple  allocations  have  occurred  on  this 
pass  (so  that  another  pass  will  be  required  to  force  lumping) . 

If  KALLLJj  =  1  and  TRNTj  >  0,  then  this  is  the  first  time  this 
device  has  been  encountered  on  this  pass,  but  it  received 
allocation  on  the  previous  pass,  possibly  at  multiple  stages, 
though  this  information  is  not  needed.  The  amount  of  allocation 
on  the  previous  pass  is  needed,  and  is  saved  as  TRNTj.  The 
algorithm  forces  these  resources  to  be  applied  at  this  stage  by 
(1)  leaving  TRNTj  unchanged  from  the  previous  pass,  (2)  updating 
the  current  performance  to  P^qw  “  PERF( -TRNTj ,  P^ow^ ' 
incrementing  the  stage  counter,  NSTAGE,  and  (4)  saving  the  stage 
device  assignment  =  I. 

If,  on  the  other  hand,  TRNTj  =  0  or  KALLOCj  >  1,  then  either 
the  incumbent  device  was  not  used  on  the  last  pass,  or  was  used 
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earlier  on  the  present  pass.  In  either  case,  the  incremental 
resources,  TRINC,  required  to  reach  from  P^ow  down  to  P^ext 
to  allocated  to  the  incumbent  device,  providing  that  there  are 
sufficient  resources  remaining.  TRINC  is  computed  using 
PINVj{p),  the  device  I  task  T  learning  curve  inverse  function 
defined  in  OPT/A42222:  TRINC  =  PINVj  ( -  PINVj  ( )  .  If 
TRNTj  +  TRINC  >  UBj,  then  there  are  insufficient  device  I 
resources  to  allow  an  additional  allocation  of  TRINC,  so  TRINC  is 
corrected  to  TRINC  =  UBj  -  TRNTj,  P^o^  is  determined  by  P^ow 
PERFj (-TRINC,  Pnow^'  TRNTj  is  updated  to  TRNTj  =  UBj. 

Otherwise,  P^ow  TRNTt  are  updated  according  to  Pnow  =  ^next 
and  TRNTj  =  TRNTj  +  TRINC. 

At  the  end  of  this  activity  the  stage  count  NSTAGE  is 
incremented  and  the  corresponding  device  assignment  is  made: 
IDVnstaGE  “ 

OPT/A422225:  Determine  Next  Algorithm  Action.  If  P^ow  ~ 

PgNTRY  (within  roundoff  error)  then  the  pass  is  complete. 
Otherwise,  training  is  still  needed  to  bridge  the  gap  between 
Pentry  ^now'  control  is  transferred  either  to  OPT/A422222 

or  to  OPT/A422223.  Control  is  transferred  to  OPT/A422222  if  the 
allocation  Just  made  was  in  some  way  constrained,  and  did  not 
correspond  to  reaching  the  economic  transfer  level  PX j j  for  some 
device  j;  this  case  is  flagged  by  INEXT  having  been  left  at  an 
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initial  value  ineligible  as  an  index  of  a  candidate  device) .  If 
the  allocation  did  correspond  to  an  economic  transfer  to  an 
suitable  next  device,  then  control  is  transferred  to  OPT/A422223, 
after  updating  the  incumbent  to  I  =  INEXT. 

If  the  pass  is  complete,  then  the  flag  ALLOCM  is  examined  to 
see  whether  multiple  allocations  were  made  to  the  same  device. 

If  so,  then  control  is  transferred  to  0PT/A422221  to  begin 
another  pass.  Otherwise  the  solution  is  complete.  In  completing 
the  activity  some  minor  "housekeeping"  is  done,  including 
reversing  the  array  IDV^,  ....  so  that  the  numbering 

will  correspond  to  training  from  entry  performance  to  standard, 
rather  than  the  "backward"  order  used  in  the  single-task 
optimization. 

If  the  number  of  passes  becomes  excessive,  then  the  problem 
is  deemed  insolvable,  and  OPT/A42222  is  abandoned,  with  the  flag 
CONDTN  set  to  indicate  that  no  solution  was  found. 


210 


Number  of 

Combinations 

0421A 


Cost  > 
Curve 
Parameters 
0421B 


Rank  Cost  Curve 
Combination  by 
Minimum 
Potential  Cost 


1 


jMinimum  Potential  Costs 
04221B 


Efficient  Devices  04A  - 

Learning  Curve  Parameters  010 


Cost 
Combo 
Ranking 
104221 A 


Entry  Performance  1AI 
Training  Stanoaro  IA2 


?e  of  Tmg  Devices 
ntry  Penormance 


Cost  Curve  Combo  Counter 

04223C  ^ 

Sureitarges 


Determine  Multi* 
Task  Resource 
Allocation  for 
Specifie  Cost  Curve 
Combination  2 


04222A 
Tentabve  Resource 
Allocation  Solution 


/ 


■T7- 


042236 


/  04223A 
Iteration  Counter 


Characterize 

'  1  1  ' 

Solubon  and  Adjust 

Surcharges  to 

Force  Constraint 

f^  '  *.1 

Compliance  3 

4 

Current  Resource  0422A 
Allocation  Solution 


Node: 

Titia:  Solve  Mulb-TsskResoufceAllocatian 

Oats: 

Number  | 

OPT/A422 

Ptobhvn 

VI/88 

OPT-34 

! 

OPT/A4223;  Characterize  Solution  and  Adjust  Surcharges. 

In  this  activity  the  aggregation  of  single-task  solutions  is 
examined  at  0PT/A42231  to  determine  if  device  resource  allocation 
constraints  are  respected,  and  whether  the  surcharges  used  to 
force  constraint  accommodation  are  acceptable.  Also,  device  cost 
surcharges  are  adjusted  in  an  attempt  to  force  constraint 
accommodation  and/or  convergence  to  acceptable  surcharge  values 
in  subsequent  iterations.  At  0PT/A42232  the  solution  is  examined 
in  terms  of  surcharge  values,  and  if  the  solution  is  acceptable 
(for  the  current  cost  curve  combination),  then  it  is  compared 
against  the  best  solution  found  to  determine  if  it  should  become 
the  incvimbent . 

0PT/A42231:  Check  for  Constraint  Violation  and  Adjust 

Surcharges .  If  TTOT^  >  TAVAILj^  for  any  device  i,  then  the 
solution  is  deemed  in  violation,  and  a  flag  VIOLIN  set  to  so 
indicate.  If  the  solution  is  not  in  violation,  then  the 
surcharges  in  current  use  are  saved,  so  that  the  solution  can  be 
reconstituted  if  needed  later. 

The  goal  of  surcharge  adjustment  is  to  have  SURCHG^  >  1  for 
any  device  i  for  which  demand  constitutes  a  problem  with  respect 
to  usage  constraints,  and  have  SURCHGj^  =  1  for  all  other  devices. 
In  this  ideal  situation  devices  are  priced  normally  unless  they 
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are  in  short  supply,  in  which  case  a  surcharge  greater  than  1  is 
applied.  The  algorithm  deems  these  constraints  to  be  satisfied, 
for  a  feasible  solution,  if  the  surcharges  and  total  allocations 
fall  within  a  fractional  tolerance  of  the  target  values.  The 
tolerance  parameter,  TLRNC ,  is  set  to  a  small  value,  say  0.05. 

Thus,  the  constraints  are  deemed  violated  if  any  one  of  the 
following  three  statements  is  true  for  any  device: 

(TTOT^  >  TAVAIL^) 
or 

(TTOT^  <  TAVAIL^  and  SURCHGi  <  1  -  TLRNC) 
or 

(TTOT^  <  ( l-TLRNC) *TAVAILi  and  SURCHG^  >  1  +  TLRNC) 

The  formulas  used  for  surcharge  adjustment  are  tied  to 
three  regions  in  SURCHGj^  x  TTOTj^  space. 

Region  1:  If  TTOTj^  >  TAVAILj^,  then  the  total  training  time 
on  device  i  is  (still)  excessive,  so  the  current  surcharge  is 
adjusted  upward  to 

SCF.CHGj^  =  (  ‘^fTRCHG^  *  TTOT^ /TAVAIL^  )  *  TWEAK 
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where  TWEAK  is  an  acceleration  parameter,  set  s-lightly  greater 
than  1,  designed  to  force  a  feasible  solution  (TTOTj^  <  TAVAILj^) 
more  quickly  than  would  occur  if  the  intuitively  appealing 
parenthetical  expression  were  used  alone  as  the  correction 
factor . 

Region  2:  If  TTOT^  <  TAVAILj^  and  SURCHGi  <  1,  then  the 
surcharge  must  be  adjusted  up  toward  1.  The  following  formula 
has  been  selected  for  this  purpose: 

SURCHG^  =  SQRT  (SURCHGi). 

Region  3:  If  TTOT^  <  TAVAIL^  and  SURCHG^  >  1,  then  the 
surcharge  must  be  adjusted  down  toward  1.  The  adjustment  should 
be  less  drastic  as  TTOTj^  nears  TAVAIL^,  so  the  following  formula 
is  employed: 

(TTOT^/TAVAIL^) 

SURGHG^  =  SURCHGj^ 

The  surcharge  adjustments  are  moderated  by  a  factor  FCA 
(fractional  change  allowed).  This  variable  is  set  to  an  initial 
value,  FCAMAX,  at  the  start  of  the  examination  of  each  cost  curve 
combination.  It  is  reduced  by  half  after  any  iteration  in  which 
TTOTj^  moves  from  one  side  of  TAVAIL^  to  the  other,  for  some 
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device  i.  This  effectively  slows  down  the  magnitude  of  SURCHGj^ 
adjustments  once  a  solution  is  approached,  thereby  damping  out 
excessive  feedback. 

OPT/A42232:  Check  for  Satisfactory  Convergence  and  Check 

for  Optimality  of  Combination.  If  the  solution  is  found  to  be 
in  violation  of  the  allocation  or  surcharge  constraints,  another 
iteration  is  begun  at  OPT/A42222. 

If  the  number  of  iterations  becomes  excessive,  the  iteration 
process  is  abandoned.  If,  in  that  case,  the  current  solution  is 
in  violation  of  the  constraints,  and  no  feasible  solution  was 
found  and  saved  on  an  earlier  iteration,  the  current  cost  curve 
segment  combination  is  abandoned.  If  the  iteration  process  must 
be  abandoned,  but  either  the  current  solution  or  a  past  solution 
is  feasible,  then  that  solution  is  evaluated  to  see  if  it 
provides  a  lower  cost  overall  than  previous  cost  curve 
combinations . 

If  convergence  is  achieved  or  iteration  is  abandoned  with  a 
feasible  solution,  then  the  solution  is  evaluated  using  the 
currently  detailed  cost  curves.  If  it  provides  a  superior 
solution,  then  the  current  cost  curve  combination  becomes  the 
Incumbent,  KBEST  =  KB,  and  the  essential  solution  elements  are 
saved,  i.e.  the  {TRNTj^},  NSTAGE^,  and  IDVj^^. 
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To  speed  convergence  and  counter  a  tendency  of  the  cost 
curves  to  underestimate  costs  when  TTOT^^  falls  outside  the 
detailing  domain,  an  ad  hoc  device  is  employed  to  give  special 
consideration  to  any  cost  curve  combination  whose  solution  falls 
within  the  current  detailing.  For  such  a  combination,  the 
estimated  cost  is  reduced  by  a  factor  1 /TWEAK.  The  true  cost 
will  be  determined  in  the  output  routine  for  the  best  solution 
found . 
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EVALUATION  OF  THE  OSBATS  MODEL  AND  SOFTWARE 

Formative  evaluation  is  a  critical  component  of  our 
development  of  the  OSBATS  DSS.  The  evaluation  effort  had  the 
following  goals: 

1.  To  determine  the  extent  to  which  the  OSBATS  model  is  relevant 
to  the  training-device  design  problems  actually  faced  by 
potential  users  at  PM  TRADE, 

2 .  To  determine  the  extent  to  which  the  data  reguired  by  the 
OSBATS  model  are  currently  available,  or  could  be  made 
available  with  reasonable  effort, 

3.  To  verify  that  the  OSBATS  software  accurately  performs  the 
functions  specified  in  the  model  documentation, 

4.  To  determine  the  ease  of  using  the  system  and  comprehending 
its  outputs, 

5.  To  determine  the  validity  of  the  predictions  made  by  the 
system. 

Our  evaluation  effort  has  consisted  of  three  types  of 
activities  to  meet  the  evaluation  goals.  The  relevance  of  the 
model,  the  data  requirements,  and  the  ease  of  use  were  evaluated 
through  interviews  with  potential  users.  These  interviews  were 
first  conducted  when  the  individual  DSS  modules  were  developed. 
The  initial  interviews  were  quite  informal,  and  were  combined 
with  a  briefing  of  the  function  of  the  module.  The  results  of 
the  initial  interviews  were  used  to  guide  the  development  of 
later  modules.  After  all  modules  had  been  developed  and 
integrated,  we  conducted  more  extensive  and  formal  interviews 
with  engineers  from  PM  TRADE.  These  interviews  evaluated  all 
modules  in  the  OSBATS  system.  Some  of  the  results  of  these 
interviews  are  being  incorporated  into  the  OSBATS  software. 

Other  results  will  provide  the  basis  for  recommendations  for 
model  and  software  enhancements. 

The  accuracy  with  which  the  software  represents  the  model  was 
verified  by  comparing  the  results  of  the  software  to  selected 
problems  with  results  calculated  by  hand.  Where  discrepancies 
were  found,  we  determined  the  source  of  the  discrepancy  and 
communicated  our  results  to  the  software-development  staff.  Our 
verification  of  OSBATS  version  1.0  found  several  discrepancies. 
Our  evaluation  of  OSBATS  version  1.1  verified  that  these  errors 
were  corrected.  We  are  now  confident  that  the  calculations  used 
in  the  software  correspond  to  the  documentation  of  the  model 
presented  in  this  report. 

The  validity  of  the  predictions  of  the  OSBATS  model  received 
the  least  emphasis  in  our  evaluation  effort.  The  main  procedure 
used  to  investigate  model  validity  was  to  conduct  several 
sensitivity  analyses,  in  which  model  inputs  were  varied  and  the 
resulting  recommendations  inspected.  These  analyses  were 


217 


conducted  to  ascertain  whether  changes  in  input  values  had 
expected  results,  and  to  discover  which  inputs  variables  were  the 
most  important  in  determining  the  model  recommendations.  The 
results  of  the  sensitivity  analyses  were  also  used  to  guide  the 
revised  research  plan. 

A  final  aspect  of  the  evaluation,  the  generality  of  the  model 
to  new  training  domains,  was  investigated  by  applying  the  CSBATS 
model  to  armor  maintenance  training.  The  details  of  that 
analysis  are  reported  in  a  later  section. 

Evaluation  of  Software  Modules 


We  conducted  a  formative  evaluation  of  the  individual  OSBATS 
prototype  modules  (three  of  five)  as  they  were  delivered  to 
ARI/PM  TRADE.  This  formative  evaluation  consisted  of  directed 
interviews  with  PM  TRADE  engineers  as  newly  delivered  modules 
were  demonstrated.  The  primary  goal  of  our  formative  evaluation 
was  to  provide  feedback  from  potential  users  to  the  staff  members 
involved  with  ongoing  development.  Specifically,  the  evaluation 
of  individual  modules  aided  OSBATS  development  in  three  ways. 
First,  information  from  the  interview-based  evaluation  of  modules 
guided  modifications  to  the  DSS  interface.  PM  TRADE  engineers 
readily  offered  opinions  on  the  useability  and  comprehensibility 
of  data  organization  and  displays.  A  summary  of  their  comments 
is  provided  below.  Second,  the  module  evaluations  aided  ongoing 
design  by  providing  us  with  information  about  standard  methods 
that  engineers  currently  use  in  concept  formulation.  Third, 
engineers  provided  feedback  on  the  correctness  or  appropriateness 
of  the  methods  used  in  OSBATS.  The  information  we  have  obtained 
in  regard  to  the  last  two  issues,  however,  has  been  limited. 
Short,  interview-based  evaluation  sessions  on  separate  modules  do 
not  allow  PM  TRADE  engineers  sufficient  time  to  comprehend  in 
detail  the  algorithms  and  general  processes  that  underlie  OSBATS. 

Results  of  the  Interview-Based  Evaluations 


Five  OSBATS  modules  were  delivered  to  ARI/PM  TRADE.  As 
mentioned  above,  HumRRO  and  ARI  staff  conducted  interview-based 
evaluations  of  the  first  three  with  PM  TRADE  engineers. 

Typically  in  these  sessions,  one  or  two  engineers  were  led 
through  the  module  and  instructed  in  its  use.  These  sessions 
were  structured  but  flexible,  allowing  for  discussion  of  data 
selection,  data  displays,  processing  methods,  and  the  mechanics 
of  the  interface.  The  data  collected  in  these  sessions  served  as 
a  record  and  fed  back  into  the  development  process.  In  this 
section,  we  summarize  the  general  recommendations  that  address 
DSS  design.  Detailed  results  have  been  presented  in  a  previous 
report  (Sticha,  Singer,  Blacksten,  Mumaw,  &  Buede,  1987) . 

The  general  recommendations  are  listed  here  under  five 
categories  that  reflect  function.  These  recommendations  largely 
represent  options  or  capabilities  that  the  engineers  would  like 
to  have  added  to  the  DSS. 
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1.  Display  format. 

a.  There  were  concerns  with  the  types  of  scales  used.  For 
example,  normalized  scales  may  read  better  as 
percentages,  and  ratio  scales  were  preferred  to 
interval  scales. 

b.  When  costs  are  shown,  they  are  usually  shown  as 
normalized  cost;  it  may  be  better  to  provide  raw  cost 
data  also.  A  multiplier  that  allows  normalized  costs 
to  be  converted  to  raw  cost  could  be  included. 

c.  When  total  cost  or  total  benefit  are  shown  and  package 
values  are  shown,  it  is  helpful  to  show  percentage 
cost. 

d.  When  developing  an  optimal  package  from  the  benefit- 
cost  curve,  it  is  useful  to  use  a  sub-optimization  to 
locate  points  on  the  curve  between  optimal  points. 

e.  Graphs  typically  show  benefit  on  vertical  axis  and  cost 
on  horizontal;  the  option  to  reverse  the  axes  should  be 
provided. 

2.  Sufficiency  of  Display: 

a.  Total  cost  and  benefit  are  useful  in  many  tables. 

b.  When  developing  optimal  packages,  an  asterisk  should  be 
used  to  mark  package  currently  being  considered. 

c.  When  using  a  derived  benefit  score,  it  is  helpful  to 
have  a  breakdown  of  its  meaning. 

d.  Need  to  have  global  model  and  data  settings  indicated. 

e.  Full  name  of  task  and  its  characteristics  should  be 
available  (in  general,  a  better  presentation  of  tasks 
is  needed) . 

f.  There  needs  to  be  an  easy  way  to  save  a  newly  defined 
task  cluster. 

3.  Getting  Around;. 

a.  A  standard  main  menu  should  be  developed,  including  the 
options  for.  Task  Library  Menu,  Examine/Modify  Task  Set 
(or  Training  Device  Set),  Display  Results,  Save 
Changes,  and  Quit. 

b.  Consistency  in  menu  item  selection  should  be 
maintained. 

c.  Consistent  method  for  moving  between  levels  and 
quitting  should  be  developed. 
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d.  Help  screens  and  instructions  are  needed  throughout. 

e.  There  is  a  need  for  an  aid  to  monitor  one's  location  in 
the  menu  hierarchy. 

4.  Aesthetics: 

a.  A  grid  (or  something  similar)  to  aid  reading  tables  and 
graphs  should  be  added. 

b.  There  is  a  need  to  provide  "clipboard"  to  save  results 
of  analyses. 

c.  On  bar  graphs,  vertical  bars  are  preferred  to 
horizontal . 

d.  More  consistent  use  of  colors  is  needed. 

e.  There  is  a  need  for  consistent  use  of  increment/ 
decrement  keys. 

5.  Confusion  Reduction: 

a.  The  keyboard  should  be  shut  off  when  the  mouse  is  being 

used. 

Use  of  the  Evaluation  Results 


The  following  guidelines,  derived  from  feedback  during  the 

interview-based  evaluation  sessions,  have  been  incorporated  into 

the  revised  version  of  the  OSBATS  software. 

1.  A  standard  main  menu  format  was  adopted. 

2.  The  capabilities  of  the  mouse  to  select  options  and  move 
between  displays  was  increased. 

3.  The  task-set-selection  screen  was  improved.  Initially,  two 
menus  were  used  to  select  a  task  set,  requiring  four  key 
strokes.  We  removed  one  menu  by  structuring  commands  more 
efficiently. 

4.  The  task-set-definition  screen  was  improved;  the  mouse  is  used 
to  exclude  or  include  tasks  for  analysis. 

5.  The  number  of  menus  was  reduced,  and  menu  structure 
simplified. 

6.  The  current  task  set  and  the  values  of  other  critical 
variables  is  now  displayed  on  all  results  displays. 

7.  The  ability  to  save  model  changes  was  improved. 
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8.  In  the  Instructional  Features  module,  normalized  cost  was 
converted  to  actual  cost. 

9.  Help  screens  were  developed  for  all  results  displays. 

10.  The  ability  to  save  a  results  screen  and  compare  results  was 
developed  for  two  modules. 

11.  Graphical  displays  were  redesigned,  and  the  use  of  color  was 
standardized  over  all  model  displays. 

12.  Longer,  more  meaningful  names  for  tasks,  training  devices, 
and  instructional  features  were  used  wherever  possible. 

Many  of  the  other  recommendations  were  considered  and  rejected, 
either  because  they  involved  major  changes  to  the  model,  or 
because  it  was  not  clear  to  us  that  these  recommendations  offered 
the  best  solution  strategy. 

Structured  Interviews  of  Potential  Users 


To  obtain  input  from  potential  OSBATS  users,  we  interviewed 
five  PM  TRADE  engineers.  Each  interview  was  structured  to 
include  a  directed  demonstration  of  OSBATS  and  a  set  of  questions 
about  its  operation.  The  interview  addressed  each  of  the  five 
OSBATS  modules,  and  each  engineer  evaluated  (a)  the  presentation 
of  data,  (b)  the  clarity  of  the  module's  results,  (c)  the 
validity  of  the  module's  approach,  (d)  the  availability  of  the 
data  required  by  the  module,  and  (e)  the  degree  to  which  all 
relevant  information  was  included  in  the  module.  As  a  group, 
these  engineers  have  been  involved  in  training-device  development 
for  an  average  of  12  years,  with  a  range  of  4  to  20  years.  Their 
average  time  with  PM  TRADE  has  been  10  years,  with  a  range  of  2 
to  20  years.  The  following  is  a  summary  of  their  comments, 
organized  by  module. 

Simulation  Configuration  Module 

Data  presentation.  Comments  were  made  on  the  presentation  of 
the  module  results  and  on  three  of  the  module  displays.  There 
were  opposing  views  about  whether  presenting  actual  costs  would 
be  more  beneficial  than  presenting  normalized  costs.  Several 
engineers  suggested  that  including  both  would  be  a  good 
alternative.  There  also  was  concern  about  being  able  to 
understand  how  costs  are  normalized.  (a)  Results  Summary  Table. 
Only  one  evaluation  indicated  that  this  table  was  useful. 

Several  engineers  had  questions  about  the  derivation  of  the  data 
in  this  table.  (b)  Task  Information  Box.  Three  of  the  five 
engineers  thought  this  display  was  necessary  and  that  it  would 
not  be  possible  to  identify  tasks  without  it.  (c)  Sensitivity 
Analysis  Display.  The  engineers  were  not  impressed  with  this 
display.  One  engineer  said  that  he  thought  a  sensitivity 
analysis  should  show  what  effect  the  cost  of  one  item  has  on  the 
cost  of  the  whole  package.  Another  engineer  commented  that  "the 
grouping  does  not  indicate  what  areas  to  look  at  for  tradeoffs." 
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These  comments  indicate  that  the  purpose  of  this  display  was  not 
clear  to  the  users. 

Module  clarity.  One  engineer  had  problems  relating  costs  to 
task  complexity.  He  commented,  "Costs  need  to  be  defined  in  some 
detail  and  they  need  better  explanation.  All  these  things  need 
examples  to  see  what's  going  on."  Another  engineer  wanted  an 
explanation  of  the  Cost-savings  presentation.  A  third  engineer 
indicated  that  he  would  like  a  "demo"  to  explain  the  purpose  of 
raising  the  slope  of  the  Cost-Benefit  Boundary  and  changing  the 
location  of  the  Task  Complexity  Line.  A  fourth  engineer  asked, 
"Why  should  I  care  about  the  boundary  line  and  how  does  it  get 
established  to  begin  with?"  The  most  obvious  points  of  confusion 
were  related  to  the  derivation  of  the  costs  and  cost  savings  and 
the  effects  of  changing  the  Task  Cluster  boundaries. 

Validity  of  approach.  The  engineers  were  asked  whether  the 
different  approaches  seemed  reasonable  and  whether  they  used  the 
same  approaches.  One  commented  that  he  liked  the  general  idea  of 
indicating  the  complexity  and  cost  of  training  tasks.  With 
regard  to  developing  the  task  sets,  the  evaluations  indicated 
that  the  approach  currently  used  is  a  task-by-task  analysis.  One 
engineer  said  that  he  does  the  same  kind  of  evaluation  "assuming 
that  a  functional  task  analysis  has  been  done."  Another  engineer 
said  he  did  not  use  that  kind  of  evaluation  but  made  estimates 
based  on  task  analysis.  In  response  to  whether  similar  tradeoffs 
are  conducted  now,  a  third  engineer  said,  "Tradeoffs  are  done  in 
terms  of  importance  of  tasks — one  versus  the  other."  However,  he 
did  indicate  that  the  OSBATS  tradeoffs  seem  reasonable.  In 
relation  to  the  cost  axis,  an  engineer  remarked,  "This  relates  to 
what  I  do  in  intuition  when  looking  for  high  cost-driver  tasks." 
In  general,  the  engineers  thought  the  approach  and  algorithms 
used  in  this  module  were  reasonable.  Their  major  concern  seemed 
to  be  the  accuracy  of  the  data  the  module  uses.  One  engineer 
remarked  that  the  validity  of  the  approach  to  cost  and  simulation 
"becomes  an  issue  of  data  acceptance." 

Data  availability.  One  response  to  the  question  of  the 
availability  of  data  was  that  the  information  is  there  but  that 
it  is  TRADOC  information  and  PM  TRADE  does  not  normally  receive 
it.  Another  response  was  that  the  information  exists  but  the 
school  hides  it.  This  engineer  said  that  the  school  is  required 
to  provide  the  information  but  hides  it  because  "they  are  not 
sure  the  data  will  get  them  what  they  want."  The  remaining  three 
engineers  answered  that  it  would  be  a  "herculean  effort"  to 
obtain  these  data.  When  asked  how  similar  this  information  is  to 
what  is  normally  available,  one  engineer  responded,  "Nothing  like 
this  is  available.  This  is  the  kind  of  info  needed  and  what  I'd 
like  to  get  but  don't  usually  have." 

Other  relevant  information.  One  engineer  thought  there  should 
be  additional  categories.  He  remarked  that  there  is  a  "need  to 
talk  about  specific  subsystems;  a  weapons  system  trainer  isn't  a 
full  mission  simulator  but  is  more  than  a  part-task  or  mission 
subset."  Only  one  engineer  suggested  a  need  for  other  task 
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cluster  information.  He  wanted  "mission  essential  infoirmation. " 
In  general,  the  engineers  were  satisfied  with  the  task  cluster 
information  provided  by  this  module. 

Instructional-Feature  Selection  Module 

Data  presentation.  There  were  few  comments  on  data 
presentation  in  this  module.  One  comment  was  that  the  only 
columns  necessary  on  the  Results  Display  are  the  Benefit,  Cost 
and  Ratio  columns.  Another  engineer  also  felt  that  the 
"percentages  are  hard  to  understand."  This  engineer  was 
referring  to  the  normalized  benefits  and  costs  that  he 
interpreted  as  percents.  This  indicates  that  additional 
explanation  of  normalization  is  needed  for  the  user  to  understand 
the  results. 

Module  clarity.  The  engineers'  comments  showed  that  the 
information  presented  in  this  module  is  not  easily  understood. 

Two  engineers  said  they  understood  the  information  though  they 
still  wanted  access  to  an  explanation,  especially  on  the 
inclusion  and  exclusion  of  weights.  The  remaining  three 
engineers  said  they  did  not  understand  the  information  presented. 
One  engineer  commented  that  he  would  like  to  have  an  explanation 
of  both  the  derivation  of  benefit  weights  and  the  normalization 
scales  used.  Another  engineer  commented,  "The  benefits  are  less 
defined  for  me  here,"  indicating  that  the  derivation  of  the 
benefits  is  not  clear. 

Validity  of  approach.  All  five  engineers  agreed  the 
instructional  feature  cost-benefit  tradeoff  is  necessary  for 
concept  formulation.  When  asked  if  the  analysis  perfoirmed  in 
this  module  is  similar  to  tradeoffs  normally  performed,  the 
answers  varied.  One  engineer  answered  that  he  tries  to  get 
TRADOC  to  assign  the  benefit  values  while  he  provides  the  costs 
and  does  the  analyses  of  cost  drivers.  Another  engineer  said 
that  he  does  it  task  by  task,  figuring  how  many  tasks  are  trained 
with  a  feature  versus  the  cost  of  the  feature.  This  engineer 
must  not  have  been  aware  that  the  instructional  feature  benefits 
are  calculated  in  a  similar  manner  in  OSBATS. 

Data  availability.  Two  of  the  engineers  indicated  that  the 
data  are  available  but  are  not  compiled  and  analyzed  the  way  they 
are  presented  in  OSBATS.  One  engineer  responded  that  the 
information  exists  but  is  not  easily  available.  The  remaining 
two  engineers,  who  happened  to  be  the  engineers  with  the  most 
experience  in  training-device  development,  indicated  that  they 
did  not  think  the  data  exist.  This  suggests  that  gathering  the 
data  necessary  to  support  this  module  may  be  difficult. 

Other  relevant  information.  Though  two  engineers  indicated 
that  training-device  development  engineers  should  be  familiar 
with  state  of  the  art  instructional  features,  other  engineers 
responded  that  they  would  like  more  information  about  the 
instructional  features  on  the  Instructional  Feature  Selection 
screen.  One  engineer  asked  for  more  information  about  the 
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various  instructional  features  because  their  names  were  not 
sufficiently  descriptive.  Another  engineer  wanted  a  list,  by 
trainer,  of  the  general  categories  of  instructional  features  that 
apply  and  felt  the  user  should  be  able  to  add  to  this  list.  In 
addition,  two  engineers  mentioned  that  the  school  sometimes 
indicates  which  instructional  features  need  to  be  included  in  the 
trainer.  In  this  case  the  user  might  want  to  be  able  to  enter 
the  school  requirements  into  the  system  as  the  base  package. 

Fidelity  Optimization  Module 

Data  presentation.  Three  displays  were  addressed:  (a)  One 
engineer  was  curious  why  the  user-defined  package  feature  could 
not  be  expressed  in  terms  of  descriptions  or  benefits.  (b)  One 
engineer  said  he  would  like  to  see  the  change  in  benefit  and  cost 
for  each  step  when  incrementing  the  Package  Display.  (c)  Several 
of  the  engineers  seemed  in  favor  of  a  comparison  screen. 

Module  clarity.  The  engineers  were  asked  if  it  is  easy  to 
understand  what  the  benefit  numbers  mean.  Only  two  of  the 
engineers  answered  yes,  and  one  of  them  said  he  understood  only 
after  receiving  an  explanation.  The  remaining  three  engineers 
answered  no.  One  engineer  suggested  incorporating  an  on-line 
explanation  on  the  Constrain  Fidelity  Dimension  screen  that  shows 
where  the  benefits  come  from.  Problems  with  the  Package  Display 
screen  seem  to  stem  from  the  data  values  rather  than  the 
capabilities  of  OSBATS.  For  example,  one  engineer  commented  that 
having  the  increment/decrement  feature  to  clarify  the  order  of 
selected  packages  "is  wonderful  if  you  believe  these  data. 

Problem  is  having  to  tailor  the  data  for  each  problem  set." 
Another  engineer  said  of  the  same  feature,  "I'm  sure  it  will  help 
you  on  your  misguided  way.  The  reason  I  say  that  is  that  I  don't 
agree  with  your  benefit  and  cost."  The  engineers  seemed  to  have 
the  most  difficulty  with  comprehending  what  the  benefit  numbers 
meant,  and  with  accepting  the  module  data. 

Validity  of  approach.  Three  engineers  had  problems  with  the 
rationale  behind  the  determination  of  benefit  for  the  fidelity 
levels.  One  engineer  commented,  "My  problem  is  that  you  are 
sometimes  trying  to  see  something  that's  smaller  than  can  be 
displayed.  I  may  not  be  interested  in  this  question  because 
state  of  the  art  can  give  resolution  at  2  km,  so  300  meters  is 
irrelevant."  Another  engineer  remarked  that  he  had  some  problem 
with  the  data  also.  Though  most  of  the  engineers  did  not 
understand  the  benefit  numbers,  they  indicated  that  they 
understood  the  rules  and  algorithms  used  in  this  module.  For 
example,  one  engineer  responded,  "Your  explanation  is  easy  to 
understand  but  not  from  using  the  system."  It  is  not  clear  how 
he  came  to  understand  these  calculations. 

Data  availability.  Each  engineer  was  asked  if  the  data  or 
information  required  by  this  module  is  normally  provided  by  the 
school.  Three  of  the  engineers  answered  no.  One  engineer  said 
that  the  school  does  not  provide  the  "fidelity  stuff."  The  other 
engineer  said  the  school  is  weak  with  cost  data  but  "provides 
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fidelity  and  task  characteristics  within  reason.”  These 
responses  indicate  that  the  schools  either  do  not  provide  this 
information  or  are  inconsistent  in  providing  it  to  the 
development  engineers. 

Other  relevant  information.  The  major  concerns  in  this 
cacegory  were  the  input  and  outpuc  of  the  system.  The  engineers 
were  asked  whether  the  set  of  fidelity  dimensions  and  levels  used 
in  this  module  is  comprehensive.  Two  of  the  engineers  did  not 
feel  qualified  to  answer.  The  remaining  three  engineers  agreed 
that  the  list  is  not  comprehensive  for  general  pilot  tasks.  For 
example,  one  engineer  said  it  is  not  clear  what  the  list  is 
addressing.  He  said,  "It  covers  general  categories,  but  you 
don't  have  enough  information  to  tell  what  the  picture  under 
consideration  is."  With  regard  to  output  information,  one 
engineer  wanted  the  module  to  provide  information  on  subsystem 
designs.  Also,  incorporating  school  requirements  such  as 
schedules  into  the  system  is  important  to  the  engineers.  Another 
engineer  suggested  a  time-line  of  training-device  development 
because  certain  fidelity  dimensions  take  longer  to  develop  and 
time  is  a  realistic  constraint.  Finally,  two  engineers  mentioned 
wanting  to  see  the  "training  effect"  or  "flow  diagram  of  impact 
on  system." 

Training-Device  Selection  Module 

Data  presentation.  Engineers  were  asked  to  evaluate  the  value 
of  the  graphs  in  this  module.  The  first  graph  presented,  called 
Graph  This,  shows  the  data  presented  in  the  Results  Summary 
Table.  rht«^e  of  the  engineers  thought  the  graph  was  better  for 
concept  fo’'mulation  documentation  than  the  table.  The  only 
concern  was  that  the  graph  is  more  difficult  to  reproduce. 

Tables  are  easier  to  print.  The  Individual  Task  graphs  were 
considered  less  useful.  Three  of  the  engineers  either  did  not 
understand  the  purpose  of  the  graph  data  or  did  not  see  a  need 
for  it. 

Module  clarity.  The  areas  of  confusion  in  this  module  were 
the  Individual  Task  tables  and  graphs.  One  engineer  remarked 
about  a  task  graph,  "What  is  this  showing  me?  It  doesn't  look 
worth  it  to  use  the  device  for  such  a  short  time.  It  would  be 
more  effective  to  use  the  actual  equipment.  This  doesn't  give  me 
an  increase  in  benefit  over  the  actual  equipment."  Another 
engineer  also  said  that  the  task  graph  indicated  to  him  that 
actual  equipment  "is  best."  The  engineers  seemed  to  interpret 
the  task  graph  as  a  graph  comparing  devices  rather  than  showing 
the  effects  of  a  training  system.  More  explanation  is  probably 
necessary  for  the  Individual  Task  table  and  graph  displays. 

Validity  of  approach.  Four  of  the  engineers  said  that  the 
analysis  done  in  this  module  is  similar  to  what  they  do.  There 
were  a  few  suggestions,  however,  worth  noting.  One  engineer 
remarked,  "This  only  addresses  inventory  that  can  address  part  of 
tasks.  What  should  be  addressed  is  whether  the  tasks  not 
addressed  by  existing  trainers  are  worth  the  cost  of  developing  a 
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new  device,  like  the  Hardman.”  Another  engineer  said  he  would 
like  an  “interactive  dialogue,”  where  the  user  enters  system 
requirements  and  the  display  provides  the  design  features 
necessary  to  get  the  desired  results  in  training. 

Data  availability.  Two  of  the  engineers  thought  the  data 
required  by  this  module  would  be  very  difficult  to  get.  Two 
other  engineers  thought  the  data  are  generally  available,  but 
they  would  have  problems  gathering  and  organizing  it.  Several  of 
the  engineers  said  they  have  this  kind  of  data  available  to  them 
during  the  concept  formulation  process. 

Other  relevant  information.  One  engineer  commented  that  he 
considers  the  number  of  students  to  be  related  to  the  student- 
to-instructor  ratio,  which  is  not  a  factor  in  OSBATS.  Two 
engineers  remarked  that  the  life  cycle  is  usually  considered  to 
be  20  years  rather  than  10  years.  When  asked  about  additional 
output,  several  engineers  wanted  time  schedules  for  development 
and  delivery. 

Resource  Allocation  Module 

Data  presentation.  The  engineers  were  asked  what  they  thought 
about  the  accessibility  of  the  screens  in  this  module.  The 
responses  were  negative.  One  suggestion  was  for  a  pull-down 
menu.  Another  engineer  said  he  likes  the  Macintosh  computer  and 
its  features.  A  third  engineer  simply  responded  that  the 
organization  could  be  optimized. 

Module  clarity.  The  results  of  this  module  are  presented  in  a 
format  similar  to  the  format  of  the  Training-Device  Selection 
Module.  Comments  about  the  tables  and  graphs  made  in  that  module 
apply  to  both  modules. 

Validity  of  approach.  Four  of  the  five  engineers  said  that 
the  analyses  performed  in  the  Resource  Allocation  Module  are 
similar  to  the  analyses  they  perform.  One  engineer  stressed  that 
he  always  does  several  iterations  when  designing  a  training 
device.  He  may  not  have  realized  that  OSBATS  can  be  used  in  an 
iterative  fashion. 

Data  availability.  The  engineers  indicated  that  the  number  of 
students  required  in  training  per  year  is  always  provided  by  the 
schools.  With  respect  to  the  availability  of  the  data  necessary 
for  the  module,  two  of  the  engineers  felt  the  data  would  not  be 
available.  The  remaining  three  engineers  said  "sometimes,” 
"normally”  and  "usually”  the  data  is  available. 

Other  relevant  information.  A  number  of  constraints  were 
mentioned  that  were  not  incorporated  into  OSBATS:  (a)  size  of  the 
door  in  the  hangar,  (b)  transportability  of  the  device,  (c)  power 
cycle,  (d)  cost  schedules,  (e)  school-imposed  constraints,  (f) 
student-to-instructor  ratio,  (g)  floor  space,  (h)  physical 
building  changes,  and  (i)  operational  hours. 
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General  System  Features 


The  engineers  were  asked  to  evaluate  the  Help  and  User  Comment 
system  features.  Each  evaluation  contained  specific  ideas  about 
what  a  help  screen  should  include.  One  engineer  said  that  he 
would  like  an  "index  of  help,"  with  the  details  of  the  cost 
elements  and  Indication  for  Simulation  explained.  Another 
indicated  that  the  help  function  "should  come  as  close  to  a 
turnkey  operation  as  possible  —  completely  explain  functions  and 
options."  There  was  a  suggestion  that  the  help  screen  should 
allow  selection  of  the  level  of  instructional  detail  the  user 
desires.  There  was  also  a  comment  that  the  screens  should 
include  larger  type,  color  graphics  and  less  text.  Four  of  the 
five  engineers  thought  the  User  Comment  feature  was  a  useful 
feature.  However,  one  of  the  four  commented  that  not  having  the 
screen  to  look  at  as  a  reference  when  making  comments  hinders 
good  note-making. 

Comments  pertaining  to  general  aesthetics  of  the  system  were 
also  addressed.  They  included  a  suggestion  to  increase  the  size 
of  the  menu  bars  so  that  there  is  space  around  each  label .  There 
were  also  comments  such  as  "problems  with  the  arrow,"  "need  a 
brighter  mouse"  and  "feedback  on  mouse"  from  three  different 
engineers,  which  seems  to  indicate  that  there  was  difficulty 
keeping  track  of  and  controlling  the  cursor. 

Summary 

Many  of  the  comments  made  by  the  engineers  represent  general 
areas  of  concern  that  should  be  considered  in  further  OSBATS 
development.  Descriptions  of  these  general  areas  follow: 

1.  OSBATS  does  not  accommodate  school  requirements  as  constraints 
in  the  training-device  development  process.  For  instance,  in 
the  Instructional  Features  module,  several  engineers  mentioned 
that  the  school  often  specifies  certain  requirements.  Levels 
of  instructional  feature  packages  might  be  more  beneficial  if 
the  user  could  enter  into  the  system  the  features  that  are 
specified  by  the  school  and  should  be  included  in  the  base 
package . 

2.  A  major  concern  is  the  cost  data  and  the  cost-savings 
associated  with  simulators.  Engineers  like  cost  comparison 
charts.  Several  engineers  indicated  that  training-device 
design  is  a  process  driven  by  cost,  time  schedules  and  school 
requirements . 

3.  Development  time  seems  to  be  an  important  issue  that  is  not 
addressed  in  OSBATS.  Time  schedules  are  often  constraints  in 
the  current  training-device  development  process. 

4.  Engineers  are  confused  by  the  derivation  of  the  benefits,  both 
instructional  feature  benefits  and  fidelity  dimension 
benefits.  More  explanation  is  necessary. 
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5.  There  is  some  confusion  about  how  the  results  are  normalized. 
Normalized  numbers  are  confused  with  percentages. 

6.  "Functional  groups"  of  tasks  are  often  used  in  the  current 
training-device  design  process  (i.e.,  tasks  for  motion 
sysLtems) .  Theie  seems  to  be  on  inclination  to  look  for 
"grouping"  and  to  design  device  components  by  function. 

7.  Several  engineers  indicated  that  training-device  design 
usually  involves  coiiCentrating  on  one  device  as  opposed  to  a 
"minimum  family  of  devices."  One  engineer  said,  "usually  the 
school  won't  consider  families.”  He  said  later,  "Our 
requirements  are  in  terms  of  a  device,  not  a  combination  of 
devices."  Another  engineer  said  that  he  usually  tries  to  come 
up  with  "a  single  design." 

8.  There  was  concern  with  the  availability  of  the  data  required 
by  OSBATS  and  the  cost  involved  with  having  to  gather  data  for 
each  different  problem  addressed.  One  engineer  asked,  "How 
usable  is  the  system?"  His  opinion  is  that  "in  its  present 
form,  not  being  flexible  and  n^^eding  lots  of  data,"  the  system 
is  limited  in  use.  Another  engineer  said  that  it  is  important 
that  the  user  be  able  to  get  into  the  system  and  "inspect  or 
change  data  for  benefits  and  cost." 

Verification  of  OSBATS  Software 

The  verification  of  the  OSBATS  software  took  the  form  of 
predict  and  verify  —  prediction  by  hand  simulation  and 
verification  by  comparison  to  the  OSBATS  output.  More 
specifically,  we  altered  values  in  the  OSBATS  input  files,  hand- 
calculated  OSBATS  outputs  using  the  correct  formula,  and  then  ran 
OSBATS  with  the  altered  input  files  to  compare  the  program's 
results.  This  was  done  for  each  output  in  each  OSBATS  module. 
Verification  was  conducted  for  both  version  1.0,  which  was 
delivered  in  July  1987,  and  version  1.1,  which  was  delivered  in 
March  1988.  The  results  of  these  evaluations  are  briefly 
described  in  the  following  subsections. 

Verification  of  Version  1.0 


The  verification  process  uncovered  several  inconsistencies 
between  the  model  and  the  version  1.0  software.  Some  of  the 
problems  came  from  the  fact  that  the  model  was  being  modified  at 
the  same  time  that  software  was  being  developed.  Thus,  for 
example,  the  Simulation  Configuration  Modules  was  using  an 
earlier  version  of  the  learning  curve  than  was  represented  in  the 
model  description,  and  implemented  in  the  other  software  modules. 
Since  individual  modules  were  developed  to  operate  independently, 
they  were  occasionally  inconsistent  in  the  methods  used.  The 
integration  of  the  modules  eliminated  these  inconsistencies. 
Finally,  some  inconsistencies  were  errors  that  were  not 
discovered  when  the  modules  were  first  developed. 
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The  results  of  the  verification  of  version  1.0  provided 
feedback  to  the  software  development  staff.  All  of  the  errors  in 
discovered  in  version  1.0  were  addressed  in  the  development  of 
version  1.1. 

Verification  of  Version  1.1 


Verification  of  version  1.1  of  the  OSBATS  software  was 
conducted  using  the  same  general  procedure  that  was  used  for 
version  1.0.  All  aspects  of  the  program  were  examined.  However, 
special  attention  was  paid  to  those  areas  that  were  newly 
developed  in  version  1.1,  or  in  which  errors  were  found  in  the 
earlier  version  of  the  software.  In  general,  the  evaluation 
determined  that  the  methods  used  by  the  software  are  correct.  An 
overview  of  the  results  is  presented,  by  module,  below. 

Simulation  Configuration.  Our  evaluation  of  the  Simulation 
Configuration  Module  verified  that  the  errors  of  the  previous 
version  were  corrected,  and  uncovered  some  other  minor 
discrepancies . 

1.  We  tested  this  module  to  verify  that  the  new  cost  curve 
formula'^  had  been  incorporated  into  this  version.  We  carried 
out  this  testing  by  comparing  Simulation  Configuration  results 
with  Training  Device  Selection  results  when  the  Training 
Device  Selection  Module  was  run  with  the  visual  meteorological 
conditions  (VMC)  trainer  and  the  actual  equipment  at  costs 
equal  to  the  variable  hourly  cost.  For  the  tasks  tested 
(chosen  randomly) ,  the  value  of  Device  Hours  Used  calculated 
in  the  Simulation  Configuration  was  comparable  to  the  time 
allocated  to  the  VMC  trainer  in  Training  Device  Selection. 

2.  The  system  calculates  the  Cost  Savings  only  once.  If  the  task 
cluster  is  changed  during  a  run  of  the  program,  the  cost 
savings  figures  from  the  previously  used  task  cluster  are 
retained. 

3.  From  the  Individual  Task  display,  the  Task  Complexity  Line  can 
not  be  adjusted.  Selecting  Task  Complexity  Line  causes  the 
line  to  become  0.1  and  remain  there. 

Instructional-Feature  Selection.  This  version  contains  the 
newly-incorporated  Instructional  Feature  Rule  Base.  We  verified 
that  the  rule  base  is  printing  its  output  file  correctly.  This 
involved  running  the  rule  base  for  a  random  ten  of  the  38  tasks 
and  verifying  that  the  results  of  the  rule  base  were  reflected 
accurately  in  the  output  file. 

Fidelity  Optimization.  This  version  of  OSBATS  contains  a 
Fidelity  Rule  Base  which  generates  the  a  matrix  of  cue  and 
response  requirements  by  task.  In  addition,  the  task  economic 
weights  are  now  calculated  by  the  system,  rather  than  being  read 
directly  from  a  file.  To  verify  that  the  benefit  and  cost  of 
each  level  of  each  fidelity  dimension  is  being  calculated 
correctly,  we  chose  one  dimension.  Visual  Resolution,  and  hand 
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calculated  the  benefits  and  costs  across  all  levels  of  this 
dimension.  We  found  that  the  task  economic  weights,  fidelity 
benefits,  fidelity  costs,  percent  benefits  and  percent  costs  are 
being  calculated  correctly. 

Training-Device  Selection.  The  calculations  in  the  Training- 
Device  Selection  module  have  changed  little  since  the  last 
version.  However,  re-verifying  these  calculations  enabled  us  to 
verify  other  calculations  performed  by  the  system. 

1.  We  chose  a  single  task.  Task  1054,  and  hand-calculated  the 
slope  of  the  learning  curves,  the  crossover  points  from 
training  on  one  device  to  another,  the  time  and  cost  of 
training  on  each  training  device  and  the  performance  levels 
achieved  on  each  device.  Our  results  were  identical  to  the 
program  results.  This  indicates  that  the  device  asymptotes 
and  time  multipliers  are  being  calculated  correctly. 

2.  We  then  went  through  the  calculations  again  to  verify  the  Re¬ 
iteration  function,  which  performs  the  calculations  using  the 
calculated  annual  utilization  (UACTj)  for  each  task  in  place 
of  the  given  nominal  utilization  (UNOMj).  The  results  of  out 
hand  calculations  were  identical  to  the  program  results. 

Resource  Allocation.  The  Resource  Allocation  Module  is  the 
most  complex  of  the  modules,  and  was  revised  more  extensively 
than  the  other  modules.  Our  evaluation  of  this  model  found 
several  problems  with  both  the  model  calculations  and  the  user 
interface. 

1.  We  found  that  the  Resource  Allocation  optimization 
calculations  do  not  necessarily  find  the  least  expensive 
solution. 

2.  When  selecting  Default  from  the  "Select  Default/Modified 
Constraints"  window,  the  Default  constraints  are  those 
constraints  specified  most  recently.  To  view  the  results  with 
no  constraints  after  constraints  have  been  used,  the  user  must 
return  to  Select  Problem  and,  in  effect,  restart  the  module. 

3.  The  name  of  user-designed  training  devices  are  not  always 
represented  correctly  by  this  module.  On  the  Show/Edit 
Constraints  and  Individual  Task  Graph  displays,  a  device  named 
"sevenmillion"  might  show  up  as  "to". 

Conclusions.  The  evaluation  effort  found  that,  in  general, 
the  software  was  making  the  calculation  correctly,  in 
correspondence  with  the  model  documentation.  Several 
discrepancies  were  found,  however.  Most  of  them  were  minor,  and 
all  have  been  corrected.  We  are  currently  confident  that  the 
calculations  performed  by  the  software  correspond  to  the  model 
description  presented  in  this  report. 
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Sensitivity  Analyses 


Sensitivity  analysis  refers  to  a  set  of  procedures  used  to 
assess  the  effects  of  manipulations  of  a  model's  variables  on  the 
results  of  the  model.  These  techniques  rely  on  making  systematic 
changes  in  the  values  of  input  data  and  measuring  the  resulting 
changes  in  model  results.  The  sensitivity  analyses  had  the 
following  two  goals. 

1.  To  determine  the  responsiveness  of  the  model  to  variable 

manipulations,  and 

2.  To  ascertain  the  validity  of  critical  model  assumptions. 

Because  sensitivity  analyses  do  not  involve  empirical  human 
performance  data,  they  do  not  provide  a  rigorous  test  of  the 
model  validity.  Rather,  they  test  validity  by  ascertaining  the 
correspondence  of  model  predictions  to  general  trends  and 
expectations.  We  describe  the  results  in  relationship  to  these 
two  goals. 

Model  Responsiveness 

We  addressed  model  responsiveness  by  varying  the  major  model 
inputs  over  a  vide  range  of  values  and  determining  the  effect  on 
the  model  recommendations.  In  most  cases,  we  used  the  results  of 
the  Training  Device  Selection  Module  as  the  main  dependent 
measure  to  assess  sensitivity.  The  following  subsections 
summarize  the  results  of  the  sensitivity  analyses. 

Task  training  requirements.  We  investigated  the  effects  of 
changes  in  entry  performance  level,  performance  standard,  and 
task  cue  and  response  requirements.  Task  cue  and  response 
requirements  are  calculated  by  the  fidelity  rule  base,  rather 
than  basic  input  data.  However,  the  task  cue  and  response 
requirements  were  used  as  the  basis  of  the  sensitivity  analyses, 
rather  than  the  task  activity  data  on  which  they  are  based, 
because  there  are  fewer  cue  and  response  dimensions  than  task 
activity  dimensions,  and  because  the  task  cue  and  response 
dimensions  are  numeric,  and  more  amenable  to  sensitivity 
analyses. 

We  found  that  lowering  the  entry  performance  causes  the  less 
expensive  devices  to  be  used  in  the  initial  training  of  a  task. 
However,  the  performance  at  which  training  switches  from  the  less 
expensive  device  to  the  more  expensive  device  (the  crossover 
point)  remains  the  same.  The  entry  performance  affects  the 
training  times  through  the  head  start  and  the  time  scaling 
factor. 

Increasing  the  performance  standard  causes  a  roughly 
proportional  increase  in  the  crossover  points.  Thus,  when  the 
performance  standard  is  increased,  the  same  training  devices  are 
generally  recommended.  This  somewhat  counter-intuitive  result 
occurs  because  the  performance  standard  is  used  in  the  definition 
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of  the  cue  and  response  requirements.  That  is,  the  cue  and 
response  requirements  are  defined  as  the  fidelity  required  to 
reach  the  performance  standard.  Tnus,  changing  the  performance 
standard  without  making  the  corresponding  change  in  the  cue  and 
respoiise  requirements  would  have  little  effect  on  the  results. 

The  performance  standard  affects  the  training  allocation  through 
the  benchmark  asymptote  and  the  time  scaling  factor. 

Task  cue  and  response  requirements  are  among  the  most  critical 
variables  in  determining  the  recommendations  of  the  OSBATS  model . 
Increasing  the  cue  and  response  requirements  causes  a  decrease  in 
the  device  asymptotes.  For  example,  for  a  task  with  many  high 
fidelity  requirements,  increasing  the  Audio  requirement  from  0.54 
to  0.90  caused  the  asymptote  to  decrease  roughly  five  percent. 

For  a  low-fidelity  task,  increasing  the  Audio  requirement  from 
0.54  to  0.90  caused  the  asymptote  to  decrease  roughly  15  percent. 
An  increase  in  the  asymptotes  causes  a  shift  in  the  training 
allocation  away  from  low-cost  devices  toward  the  sophisticated 
training  devices  and  actual  equipment. 

Other  Task  Data.  The  major  variables  that  were  investigated 
here  involve  estimates  of  training  hours  and  costs.  We  found 
that  multiplying  through  the  task  training  hours  (2A1  -  2A5)  by  a 
constant  factor  produces  a  change  in  training  times  by  exactly 
that  factor.  The  relative  proportion  of  training  time  allocated 
to  each  device  does  not  change,  however.  This  result  indicates 
that  the  devices  chosen  by  the  Training-Device  Selection  Module 
are  completely  insensitive  to  the  task  training  hours,  when  all 
five  of  the  data  elements  are  varied  in  proportion.  Changes  in 
training  hours  would  have  an  impact  on  the  results  of  the 
Instructional-Feature  Selection  and  Fidelity  Optimization 
Modules,  however,  because  they  would  affect  the  task  weights. 

We  found  that  increasing  the  setup  hours  by  a  factor  of  ten 
without  changing  other  task  training  times  causes  a  marked 
increase  in  training  times  with  respect  to  both  the  training 
devices  and  the  actual  equipment.  In  the  tasks  we  examined,  an 
increase  by  a  factor  of  ten  caused  the  training  time  to  double. 

Training  Device  Data.  The  training  device  cue  and  response 
capability  (FCAP^j)  has  a  large  effect  on  the  model 
recommendations  that  is  analogous  to  the  task  cue  and  response 
requirements.  The  effects  are  not  completely  analogous,  however, 
because  the  effect  of  FCAPjj  is  moderated  by  the  minimum 
performance  parameter,  FMINj.  We  found  that  the  change  in  FCAP^j 
has  the  largest  influence  on  the  asymptotes  when  the  minimum 
performance  parameter  is  low.  We  set  the  FCAPjj  for  all 
dimensions  equal  to  1.0  except  for  one  dimension  we  call  j.  We 
varied  the  FCAPjj  for  this  dimension  from  0.0  to  1.0.  At  FCAPjj  = 
0.0,  the  task-device  asymptote  is  FMIN-.  At  FCAPjj  =  1.0,  the 
asymptote  is  1.0.  As  FMIN:  increases  from  0.0  to  1.0,  the 
asymptotes  fall  increasingly  close  together  as  FCAPjj  is  varied. 
Thus,  the  largest  changes  occur  at  the  lowest  values  of  FCAPjj 
when  FMINj  is  also  low. 
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Fidelity  Dimension  Data.  We  examined  two  fidelity  dimension 
data  variables,  fidelity  dimension  cost  and  the  fidelity 
dimension  minimum  performance  parameters. 

We  found  that  as  the  exponent  in  the  fidelity  dimension  cost 
estimating  function  gets  very  large  (ten  times  its  original 
value) ,  the  cost  curve  becomes  increasingly  less  linear.  Low 
levels  of  technology  are  still  cheap.  Higher  levels  of 
technology  get  dramatically  more  expensive. 

Lowering  the  minimum  performance  parameters  for  all  11 
fidelity  dimensions  to  zero  causes  a  decrease  in  training  on  the 
relatively  unsophisticated  devices  such  as  the  NOE  trainer. 
Increasing  the  FMINj  for  all  eleven  dimensions  to  0.90  causes  a 
shift  in  allocation  of  training  towards  the  Weapons  and  NOE 
trainers  and  away  from  the  VMC  trainer.  We  concluded  that 
increasing  the  FMINs  benefits  the  low-cost  trainers  at  a  greater 
rate  than  the  high-cost  trainers. 

Training  System  Data.  We  examined  three  training  system 
variables,  the  maximum  instructional  feature  effect,  the  assumed 
setup  savings  percentage  and  the  maximum  num.ber  of  instructional 
features. 

We  feel  reasonably  confident  that  SUSAV,  the  setup  savings 
percentage,  is  greater  than  or  equal  to  50%  but  less  than  or 
equal  to  80%.  We  also  feel  confident  that  IFBN,  the 
instructional  feature  benefit,  has  a  value  greater  than  or  equal 
to  0%  but  less  than  or  equal  to  50%.  Therefore,  the  time 
multiplier  varies  from  1.0  to  2.73  as  IFBN  and  SUSAV  vary.  At  a 
glance  it  would  seem  that  this  fuzziness  would  have  enormous 
implications  on  the  learning  curve.  However,  it  is  only  in  the 
first  few  hours  of  training  that  the  time  multiplier  affects  the 
learning  curve  more  than  the  asymptote. 

We  feel  reasonably  confident  that  the  MAXIF,  the  maximum 
number  of  instructional  features  that  benefit  training,  is 
between  2  and  5.  Thus,  the  largest  amount  of  fuzziness  in  the 
time  multiplier  is  approximately  0.06'  comparatively  little. 
Therefore,  the  fuzziness  associated  with  the  value  of  MAXIF  has 
very  little  affect  on  the  OSBATS  model. 

Implications  of  Analyses  on  Validity 

Several  of  the  results  of  the  sensitivity  analyses  help 
confirm  the  validity  of  the  OSBATS  model,  in  that  they  correspond 
to  our  expectations.  Other  results  offer  important  characteriza¬ 
tions  of  the  model  that  would  provide  the  basis  for  a  critical 
test  of  model  assumptions.  The  following  are  some  of  the  more 
important  results,  restated  in  somewhat  more  general  terms. 

1.  Simpler  training  devices  are  more  appropriate  at  lower  skill 

levels;  higher  fidelity  training  devices  are  more  appropriate 

at  higher  skill  levels. 
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2.  The  most  critical  process  in  the  model  application  is  the 
process  that  determines  task  cue  and  response  requirements. 

3.  The  model  predicts  that  fidelity  can  be  sacrificed  more 
readily  on  less  critical  fidelity  dimensions  (with  higher 
FMINj)  than  it  can  on  more  critical  fidelity  dimensions. 

4.  Fidelity  is  more  important  than  instructional  features  at  high 
skill  levels;  instructional  features  are  more  important  at  low 
skill  levels,  although  it  is  difficult  to  say  at  what  skill 
level  fidelity  becomes  more  important  than  instructional 
features. 

5.  Fidelity  requirements  are  related  to  the  performance  standard. 
If  the  standard  is  raised,  then  it  may  require  greater 
fidelity  to  train  to  that  standard. 

6.  The  total  training  hours  required  to  train  a  task  do  not 
effect  which  training  devices  are  selected  for  that  task,  but 
it  does  affect  the  total  cost  of  training. 

Most  of  these  results  are  expected,  and  serve  to  give  us  some 
confidence  about  the  validity  of  the  OSBATS  model,  albeit  on  a 
very  informal  level.  Other  results,  particularly  those  that 
relate  the  importance  of  fidelity  and  instructional  features  and 
tie  fidelity  requirements  to  the  performance  standard,  provide 
the  opportunity  for  critical  tests  of  the  OSBATS  model. 
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EXAMPLE  APPLICATION  OF  OSBATS 


The  OSBATS  system  is  composed  of  five  modules  developed  in  the 
C  programming  language,  and  two  rule  bases  developed  using  EXSYS 
(Copyright  Exsys,  Inc.),  an  expert  system  shell.  Installation 
of  OSBATS  requires  an  IBM  PC/AT,  Zenith  248  or  compatible  with 
640K  of  memory  and  a  10  megabyte  hard  disk.  Additional  hardware 
requirements  include  an  enhanced  graphics  adapter  (EGA)  and  color 
monitor,  an  80287  numeric  coprocessor,  and  a  Microsoft-compatible 
mouse. 

The  user-interface  consists  of  a  variety  of  menus  and 
displays.  After  the  software  is  installed  and  OSBATS  has  been 
started,  the  user  interacts  with  the  software  using  the  mouse. 

The  various  modules  can  be  accessed  through  the  Module  Menu. 
Likewise,  each  module  has  a  menu  of  its  own  outlining  its 
subsections.  Each  module  subsection  contains  a  display  or  series 
of  displays.  A  display  may  be  a  graph  or  a  table  of  results  or 
it  may  be  for  the  selection  of  constraints  or  for  saving  results. 
The  user  can  access  various  displays  within  a  module  subsection 
through  a  menu  at  the  bottom  of  each  display.  OSBATS  also 
includes  a  HELP  feature  to  assist  the  user,  which  can  be  accessed 
from  any  display.  The  figures  presented  in  this  section  resemble 
the  displays  produced  by  OSBATS,  except  that  the  menus  are 
omitted . 

This  section  describes  an  application  of  the  OSBATS  model  to 
an  artificial,  but  realistic  problem  related  to  Army  Aviation 
training.  The  goal  of  the  description  is  to  illustrate  both  the 
kinds  of  solutions  to  concept-formulation  problems  provided  by 
the  OSBATS  model  and  the  ways  that  the  individual  modules 
interact  to  produce  these  solutions.  The  example  should  provide 
contextual  information  to  help  the  reader  understand  the  specific 
calculations  presented  in  the  formal  model  description.  This 
section  will  not  describe  the  calculations  in  any  detail. 

Rather,  it  will  concentrate  on  how  the  results  of  different 
calculations  can  be  integrated  to  provide  overall  support  to  the 
concept-formulation  process.  The  results  presented  in  this 
section  were  calculated  by  the  decision  support  software  that 
implements  the  OSBATS  model.  User  and  programmer  documentation 
for  this  software  are  presented  in  separate  reports. 

The  sample  problem  is  based  on  an  existing  training  system, 
the  AH-1  Airman  Qualification  Course  (AQC) .  The  section  begins 
with  brief  descriptions  of  the  training  requirements  for  the 
course,  and  the  data  collection  methods.  The  section  continues 
with  an  overview  of  the  sample  analysis.  The  OSBATS  model  is  a 
flexible  set  of  tools  that  can  be  tailored  to  the  needs  of  the 
problem  and  the  preferences  of  the  user.  In  the  example  problem 
we  have  selected  a  minimal  analysis  that  illustrates  the  use  of 
each  module.  Following  the  analysis  overview,  we  discuss  the 
operation  of  each  of  the  modules.  Finally,  we  summarize  the 
results  of  the  analysis  and  describe  their  implications  for  this 
hypothetical  problem. 
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Description  of  the  Example  Problem 


To  exercise  the  models  developed  during  this  project,  and  to 
evaluate  the  model’s  data  requirements,  it  was  necessary  to 
develop  a  training  data  base  for  a  device-based  training  system. 
We  chose  the  AH-1  AQC  to  form  the  basis  of  the  initial  model 
application. 

The  AH-1  AQC  is  a  training  course  designed  to  qualify  aviators 
to  fly  the  AH-1  helicopter — one  of  the  two  attack  helicopters 
presently  in  the  Army's  aircraft  inventory.  The  AH-1  AQC  is  an 
Institutional  training  course  taught  only  at  the  U.S.  Army 
Aviation  Center,  Fort  Rucker,  Alabama.  During  the  6.4  weeks 
normally  required  to  complete  the  course,  students  receive  33 
hours  of  flight  instruction  in  the  aircraft,  3  hours  of 
instruction  in  the  AH-1  Flight  and  Weapons  Simulator  (AHIFWS) , 
and  academic  instruction  on  aircraft  systems,  emergency 
procedures,  and  tactics.  Although  some  Army  aviators  enter  the 
AH-1  AQC  immediately  after  completing  Initial  Entry  Rotary  Wing 
(lERW)  training,  it  is  more  common  for  aviators  to  enter  AQC 
training  after  completing  one  or  more  flying  duty  assignments  in 
another  aircraft.  In  either  case,  all  aviators  will  have  been 
trained  and  become  qualified  in  two  or  more  other  aircraft  prior 
to  entering  the  AH-1  AQC. 

Overview  of  Data  Collection 


The  procedures  used  to  develop  and  collect  the  data  on  the  AH- 
1  AQC  were  described  in  detail  by  Sticha,  Blacksten,  Buede,  and 
Cross  (1986).  In  general,  the  categories  of  data  that  were 
collected  met  the  requirements  of  the  OSBATS  model,  and  included 
the  following  classes  of  information: 

1.  a  comprehensive  list  of  AH-1  AQC  training  tasks, 

2.  estimates  of  AQC  students'  pre-  and  post-training  skill  level 
on  each  training  task  (data  base  category  lA) , 

3.  ratings  of  the  extent  to  which  safety  and  training  efficiency 
considerations  would  favor  training  in  a  simulated  environment 
(category  IB) , 

4.  estimates  of  the  amount  of  time  and  number  of  practice 
iterations  required  to  master  each  training  task  in  the 
aircraft  (category  2A) , 

5.  ratings  of  the  extent  to  which  several  specific  information¬ 
processing  activities  are  recpiired  to  perform  each  task 
(category  2B) ,  and 

6.  assessments  by  task  of  the  activities  that  require  high 
fidelity  in  training-device  components  to  achieve  effective 
training  (category  2C) , 
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7.  definition  of  a  family  of  generic  training  devices  and 

judgments  about  the  AH-1  AQC  tasks  that  can  be  trained  in  each 

device  (category  3) . 

The  data  that  are  used  in  this  analysis  are  shown  in  Appendix 
A,  with  asterisks  indicating  the  tasks  used  in  the  example 
problem.  The  following  paragraphs  provide  a  brief  discussion  of 
the  tasks  and  training  devices  that  were  used  in  the  analysis. 

AH-1  Training  Tasks 

The  procedure  used  to  generate  tasks  resulted  in  a  list  of  188 
individual  tasks  that  AH-1  aviators  must  learn  to  perform.  This 
list  includes  both  tasks  that  are  trained  during  the  AH-1  AQC 
(Institutional  training)  and  tasks  that  are  trained  only  after  an 
AH-1  aviator  has  been  assigned  to  an  operational  field  unit 
(Continuation  training) .  Of  the  125  tasks  that  are  trained  in 
the  AH-1  AQC  we  selected  38  tasks  to  form  the  basis  of  the 
example  problem.  These  tasks  were  chosen  to  satisfy  the 
following  conditions;  ^a)  all  tasks  were  actually  trained  in  the 
AQC,  rather  than  in  continuation  training,  (b)  tasks  represented 
a  variety  of  skills,  such  as  normal  flight,  malfunction 
procedures,  and  attack  mission  tasks,  (c)  tasks  appeared  to  have 
a  variety  of  cue  and  response  requirements,  from  low  to  high 
fidelity,  and  (d)  tasks  could  not  be  trained  to  standard  using 
academic  training  only.  The  tasks  that  were  chosen  are 
enumerated  in  Table  7.  The  table  lists  the  number  of  the  task  in 
the  Aircrew  Training  Manual  (ATM) ,  the  task  name,  and  an 
abbreviated  name  used  to  refer  to  the  task  later  in  the  analysis. 

Family  of  Training  Devices 

One  of  the  primary  functions  of  the  OSBATS  model  is  to  aid  in 
defining  the  most  cost-effective  set  of  training  devices  for  a 
training  system.  To  evaluate  the  models*  utility  for  this 
purpose,  it  was  necessary  to  define  a  family  of  potential 
training  devices  that  covers  a  wide  range  of  cost  and  training 
functions  and  to  make  a  subjective  assessment  of  the  tasks  that 
can  be  trained  in  each  of  the  devices.  Working  in  concert,  the 
SMEs  developed  descriptions  of  five  training  devices  that  they 
judged  to  have  potential  value  for  training  aviators  attending 
the  AH-1  AQC;  a  Cockpit  Procedural  Trainer,  a  Doppler  Navigation 
Trainer,  an  Instrument  (IMC)  Flight  Trainer,  a  Weapon  Systems 
Trainer,  and  a  Flight  and  Weapons  Simulator.  The  Flight  and 
Weapons  Simulator  is  the  only  one  of  the  five  devices  that  has 
been  developed  for  the  AH-1  aircraft,  although  similar  devices 
have  been  developed  for  other  types  of  rotary  wing  aircraft.  The 
assumed  design  of  each  training  device  was  specified  by 
developing  tables  that  show; 

1.  the  type  of  cockpit  displays, 

2.  the  type  of  engine  instruments  (if  any), 

3.  the  type  of  cockpit  controls. 
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Table  7  Tasks  Selected  for  Example  Problem 


Task 

Task  Name 

Num. 

Abbreviation 

Task  Name 

1003 

T/0  CHECKS 

Start,  Runup,  Takeoff  Checks 

1004 

HOVER  PWR  CK 

Hover  Power  Checks 

1005 

HOVER  FLT 

Hovering  Flight 

1006 

NORMAL  T/0 

Normal  Takeoff 

1007 

S/MAX  P  T/0 

Maximum  Performance  Takeoff 

1008 

DECEL  ACCEL 

Deceleration/Acceleration 

1009 

TRFC  PTRN 

Traffic  Pattern  Flight 

1012 

DOPPLER  NAV 

Doppler  Navigation 

1013 

PRE  LNDG  CK 

Before  Landing  Checks 

1014 

VMC  APPROACH 

VMC  Approach 

1015 

SHLW  APPR 

Shallow  Approach/Landing 

1016 

CONFIND  OPN 

Confined  Area  Operations 

1018 

TERR  FLT  T/0 

Terrain  Flight  Takeoff 

1019 

TERRAIN  FLT 

Terrain  Flight 

1021 

NOE  DECEL 

Nap-of-the-Earth  Deceleration 

1022 

TRN  FLT  APP 

Terrain  Flight  Approach 

1024 

HVR  AUTOROT 

Hovering  Autorotation 

1026 

SIM  ENG  FAIL 

Simulated  Engine  Failure 

1028 

MAN  THROTTLE 

Manual  Throttle  Operation 

1029 

NO  SCAS/SAS 

SCAS  Off  Flight 

1039 

MSKG/UNMSKG 

Masking  and  Unmasking 

1042 

MOVE  TECH 

Techniques  of  Movement 

1048 

PIN/RDG  OPN 

Pinnacle/Ridgeline  Operation 

1054 

OP  RMS 

Operate  Rocket  Mgmt.  System 

1055 

WPNS  PROC 

Weapons  Cockpit  Procedure 

1056 

OP  TURRET 

Operate  197  Turret  System 

1057 

OP  FFAR  ROC 

Operate  2.75”  FFAR  Launcher 

1058 

OP  TOW  MSL 

Operate  TOW  Missile  System 

1061 

NOE  FIRING 

Terrain  Flight  Firing  Techs. 

2000 

STD  AUTOROT 

Standard  Autorotation 

2003 

S/A  TRQ  MAL 

Simulated  Antitorque  Malfunc. 

2004 

L-A,H-S  AR 

Lo  Level  Hi  Speed  Autorotation 

3008 

DROOP  CMP  FL 

Droop  Compensator  Failure 

3010 

TAIL  ROT  FL 

Tail  Rotor  Failure  at  a  Hover 

3011 

MAST  BUMPING 

Mast  Bumping 

3025 

THROTTLE  FL 

Emergency  Shutdown — Throttle 

3030 

LNDG/TREES 

Landing  in  Trees 

3033 

SCAS_FAILURE 

SCAS  Failure 

4 .  the 

type  of  motion 

systems  (if  any) , 

5 .  the 

type  of  extra- 

cockpit  visual  systems  (if  any). 

6 .  the 

type  of  aerodynamic  models  (if  any) ,  and 

7.  the 

type  of  noise 

simulated  (if  any). 

Once  the  generic  design  characteristics  of  the  family  of 
training  devices  had  been  specified,  the  SMEs  proceeded  through 
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the  master  task  list  (described  above)  and  identified  the  tasks 
that  they  judged  could  be  trained  effectively  in  each  device. 

Analysis  Overview 

Our  concept  of  operation  for  the  OSBATS  Model  is  based  on  the 
iterative  use  of  the  five  model  tools  to  make  recommendations 
regarding  the  definition  of  task  clusters,  the  design  of  training 
devices  for  individual  clusters,  and  the  allocation  of  training 
resources  among  training  devices.  This  iterative  concept  of  use 
reflects  the  interactions  inherent  in  the  training-concept 
development,  training-device  design,  and  resource  allocation 
components  of  the  training-system  design  process.  Both  the 
subset  of  tools  that  are  used  and  the  order  in  which  they  are 
used  may  vary  depending  on  the  requirements  of  the  problem  and 
the  preferences  of  the  user.  In  addition,  a  tool  may  be  used 
several  times,  each  time  producing  a  more  refined  solution. 

For  the  purposes  of  this  example,  however,  we  apply  the  tools 
in  the  following  order: 

1.  The  Simulation  Configuration  module  defines  clusters  of  tasks 
for  actual  equipment  (AE) ,  a  full-mission  simulator  (FMS) ,  and 
part -mission  simulators  (PMSs). 

2.  The  Instructional  Feature  Module  focuses  on  the  FMS  tasks 
defined  by  the  Simulator  Configuration  Module  and  identifies 
the  instructional  features  that  produce  the  greatest 
improvement  in  training  efficiency  given  their  development 
cost. 

3.  The  Fidelity  Optimization  Module  identifies  the  areas  in  which 
investment  in  additional  fidelity  would  be  cost  efficient. 

The  module  incorporates  the  results  of  the  Instructional 
Feature  Module  to  produce  two  candidate  FMS  designs  at 
different  development  cost  levels. 

4.  The  Training-Device  Selection  Module  is  applied  to  evaluate 
training  systems  with  each  of  the  two  candidate  devices 
generated  by  the  Fidelity  Optimization  Module.  For  each  of 
these  devices,  the  Training-Device  Selection  Module  determines 
the  minimum  cost  to  meet  the  training  requirements. 

5.  The  Resource  Allocation  Module  determines  the  minimum  cost  to 
meet  training  requirements  using  the  best  training  device 
determined  by  the  Training-Device  Selection  Module.  The 
Resource  Allocation  Module  takes  into  account  hypothetical 
constraints  on  the  number  of  training  devices  that  may  be 
obtained. 

In  our  description  of  the  OSBATS  model ,  we  point  out 
alternative  approaches  that  could  be  taken  to  the  application  of 
the  model,  opportunities  for  iteration  of  the  model,  and  ways  the 
model  could  be  used  to  refine  the  solution  that  is  obtained  in 
the  analysis. 
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Analysis  Details 


The  following  sections  describe  each  of  the  modules  in  turn, 
and  illustrate  the  kinds  of  data  that  are  considered  in  each 
analysis,  the  procedures  by  which  data  are  combined  to  produce  a 
recommendation,  and  the  particular  recommendations  that  are  made 
by  each  module.  The  procedures  are  described  at  a  general  level. 
The  reader  who  wishes  a  more  detailed  description  of  the 
procedures  will  find  them  in  the  formal  model  description  in  a 
later  section.  The  goal  of  this  section  is  to  provide  a 
description  of  the  model  from  a  user's  perspective;  thus,  it 
concentrates  on  the  types  of  recommendations  produced  by  the 
model,  how  user  judgment  is  incorporated  into  the  model  results, 
and  how  the  results  can  be  used  to  develop  and  evaluate  training- 
system  designs. 

Simulation  Configuration  Module 

The  Simulation  Configuration  Module  examines  the  tasks  to  be 
trained  and  partitions  the  task  set  into  three  clusters:  (a)  a 
cluster  of  tasks  that  should  be  trained  on  a  full-mission 
simulator  (FMS)  that  represents  most  of  the  actual  equipment 
subsystems,  (b)  a  cluster  of  tasks  that  should  be  trained  on  one 
or  more  part-mission  simulators  (PMSs)  that  represent  selected 
subsystems  or  that  represent  actual  equipment  at  a  low  level  of 
fidelity,  and  (c)  a  cluster  of  tasks  that  should  be  trained  on 
actual  equipment  (AE) .  This  recommendation  is  based  upon 
simulator  requirements  for  task  training  and  the  potential 
operating-cost  savings  associated  with  simulators,  balanced 
against  simulator  development  cost,  as  estimated  by  the  task  cue 
and  response  requirements. 

Initial  model  recommendations.  In  this  example  problem,  we 
assume  that  all  relevant  data  have  been  collected  and  are  present 
in  the  OSBATS  data  base.  The  Simulation  Configuration  Module  can 
then  make  an  immediate  recommendation  when  the  user  has  specified 
the  set  of  tasks  to  be  trained.  In  the  example  problem,  the 
model  recommends  that  22  of  the  38  tasks  be  trained  on  a  full- 
mission  simulator,  11  tasks  be  trained  on  part-mission 
simulators,  and  5  tasks  be  trained  on  actual  equipment.  The 
initial  recommendations  of  the  module  are  shown  in  Table  8 . 

These  recommendations  are  based  on  the  following  assumptions: 

1.  Simulation  requirements  and  operating  cost  savings  are 
weighted  equally  in  determining  the  indications  for 
simulation,  the  measure  of  simulation  benefit  used  in  the 
module. 
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Table  8  Initial  Recommendations  of  the  Simulation  Configuration 
Module 


Actual  Part -Mission  Full-Mission 

Equipment  Simulator  Simulator 

Task  #  Name  Task  #  Name  Task  #  Name 


1012  DOPPLER  NAV  1003 

T/O  CHECKS 

1015 

SHLW  APPR 

1004 

HOVER  PWR  CK 

1016 

CONFIND  OPN 

1005 

HOVER  FLT 

1019 

TERRAIN  FLT 

1006 

NORMAL  T/O 

1042 

MOVE  TECH 

1007 

S/MAX  P  T/O 

1048 

PIN/RDG  OPN 

1008 

DECEL  ACCEL 

1054 

OP  RMS 

1009 

TRFC  PTRN 

1056 

OP  TURRET 

1013 

PRE  LNDG  CK 

1057 

OP  FFAR  ROC 

1014 

VMC  APPROACH 

1058 

OP  TOW  MSL 

1018 

TERR  FLT  T/O 

1061 

NOE  FIRING 

1021 

NOE  DECEL 

2000 

STD  AUTOROT 

1022 

TRN  FLT  APP 

2003 

S/A  TRQ  MAL 

1024 

HVR  AUTOROT 

2004 

L-A,H-S  AR 

1026 

SIM  ENG  FAIL 

3010 

TAIL  ROT  FL 

1028 

MAN  THROTTLE 

3033 

SCAB  FAILURE 

1029 

NO  SCAS/SAS 

1039 

MSKG/UNMSKG 

1055 

WPNS  PROC 

3008 

DROOP  CMP  FL 

3J11 

MAST  BUMPING 

3025 

THROTTLE  FL 

3030 

LNDG/TREES 

2.  The  total  operating-cost  savings  calculated  by  the  module  is 
used  as  the  basis  of  the  cost  boundary  that  divides  AE  tasks 
from  PMS  and  FMS  tasks. 

3.  The  task  complexity  line,  which  is  used  to  separate  FMS  tasks 
from  PMS  tasks,  is  located  at  the  halfway  mark  on  the 
normalized  cost  scale. 

In  this  hypothetical  problem,  the  user  will  examine  the 
assumptions  of  the  model  to  verify  the  accuracy  of  the  results. 
The  examination  begins  with  general  characteristics  of  the  model 
results  and  proceeds  to  more  specific  features. 

Examination  of  model  recommendations.  The  first  aspect  of  the 
results  to  examine  is  the  ordering  of  tasks  based  on  cost  and 
indications  for  simulation.  Table  9  lists  tasks  that  have 
extreme  values  on  either  cost  or  indications  for  simulation. 

This  display  is  not  presented  directly  by  the  software,  but  could 
be  generated  easily  by  the  user  from  other  displays.  We 
anticipate  that  the  user  will  be  able  to  verify  the  tasks  on  this 
list  given  his  general  knowledge  of  the  training  domain.  More 
detailed  information  would  be  available  to  the  user  if  it  were 
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Table  9  List  of  Tasks  with  Extreme  Values  of  Cost  or 
Indications  for  Simulation 


Tasks  with  High  Indication  for  Simulation 

1061  NOE  FIRING 
3008  DROOP  CMP  FL 

3010  TAIL  ROT  FL 

3011  MAST  BUMPING 
3025  THROTTLE  FL 
3030  LDNG/TREES 

Tasks  with  Low  Indication  for  Simulation 


1012 

DOPPLER  NAV 

1013 

PRE  LNDG  CK 

Tasks 

with  High  Cost 

1012 

DOPPLER  NAV 

1042 

MOVE  TECH 

3033 

SCAS  FAILURE 

Tasks 

with  Low  Cost 

1003 

T/0  CHECKS 

1013 

PRE  LNDG  CK 

1055 

WPNS  PROC 

3011 

MAST  BUMPING 

3025 

THROTTLE  FL 

needed  to  provide  a  general  verification  of  the  cost  and 
indicat ions-for-simulat ion  scales. 

The  module  results  shown  in  Table  9  indicate  that  six  tasks 
require  simulation.  An  examination  of  these  tasks  indicates  that 
this  result  is  reasonable;  these  tasks  involve  serious 
malfunctions  (droop  compensator  failure  or  tail  rotor  failure) , 
require  use  of  expensive  ammunition  or  large  training  areas 
(terrain  firing) ,  or  involve  dangerous  emergency  procedures 
(landing  in  trees) .  The  two  tasks  for  which  indications  for 
simulation  has  the  lowest  value  also  seem  reasonable.  These 
tasks  involve  basic  maneuvers  that  can  be  learned  relatively 
quickly  with  little  chance  of  danger. 

The  cost  of  simulation  reflects  the  cue  and  response 
requirements  of  the  tasks.  The  tasks  with  the  highest  cost 
require  a  sophisticated  visual  system,  a  sophisticated  motion 
platform  or  a  large  data  base  to  adequately  train  navigation  and 
movement  techniques.  The  tasks  with  lowest  cost  are  generally 
procedural,  and  could  be  trained  on  a  cockpit  procedures  trainer 
with  low  fidelity. 
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The  preliminary  examination  of  the  results  indicates  that  the 
cost  and  indication-for-simulation  scales  reasonably  reflect  the 
actual  cost  and  benefits  of  simulation  for  the  subject  tasks.  In 
the  next  phase  of  the  analysis,  the  user  examines  the  cost 
boundary  and  the  task  complexity  line.  The  cost  boundary 
separates  tasks  assigned  to  the  AE  cluster  from  tasks  assigned  to 
the  FMS  and  PMS  clusters.  The  task  complexity  line  separates  the 
PMS  from  the  FMS  task  clusters.  That  investigation  proceeds  by 
examining  tasks  near  these  boundaries  to  determine  whether  they 
appear  to  be  in  the  correct  task  cluster.  Making  these  close 
comparisons  requires  the  user  to  examine  the  data  that  underlie 
the  cost  and  benefit  assessments. 

Table  10  illustrates  the  data  and  intermediate  results  that 
underlie  the  overall  module  results.  The  user  has  access  to  this 
information  for  each  task  and  could  verify  that  for  most  of  the 
tasks  there  is  essentially  no  cost  savings  associated  with 
simulator  training,  though  there  may  be  a  moderate  value  for  the 
indication  for  simulation.  The  exceptions  to  this  are  tasks 
1055,  1057  and  1061,  which  have  a  high  cost  savings.  In  light  of 
these  data,  the  user  might  make  one  of  following  judgments. 

1.  The  moderate  value  of  the  indication  for  simulation  for  many 
of  the  tasks  may  lead  the  user  to  accept  the  initial  result  of 
the  module,  that  most  of  the  tasks  be  trained  in  a  simulator. 


Table  10  Data  Underlying  Results  for  Task  #  1057;  Operate  2.75" 
FFAR  Launcher 


Weights;  Training  Requirements  Index;  0.5 
Operating-Cost  Savings  Index;  0.5 


Relative  Cost 

Cue  &  Response 
Dimension 

=  0.88 

Rqmt.  Cost 

Indication  for  Simulation  =  0.20 

Factor  Value  Wt. 

Visual  Resol. 

0.98 

2638 

Training  Rqmt  Index 

1.00  0.50 

Visual  Content 

0.58 

496 

Visual  Texture 

0.84 

1108 

Absolute  Requirement 

no 

Visual  Front 

0.86 

403 

Special  Conditions 

3  of  3 

Visual  Side 

0.46 

10 

Training  Effects 

2  of  2 

Point  Effects 

0.00 

504 

Area  Effects 

0455 

147 

Ops  Cost  Saving  Index 

0.64  0.50 

Platform  Mot 

0.00 

0 

Seat  Motion 

0.36 

67 

Ops  Cost  Savings  $2505  /student 

Sound  Effects 

0.54 

96 

AE  Hours  Saved 

1.41 

Map  Size 

0.58 

634 

Device  Hours  Used 

10.28 

Total  Cost 

6106 
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2 .  The  user  may  think  that  tasks  with  low  cost  savings  should  be 
trained  on  actual  equipment.  The  user  might  then  raise  the 
weight  given  to  the  operating-cost  savings  index  in  the 
determination  of  the  indication  for  simulation,  to  make  cost 
savings  a  more  important  consideration  in  the  module. 

3.  The  user  may  determine  that  for  tasks  with  low  cost  savings 
the  training  requirements  justify  simulator  training.  The  user 
might  then  raise  the  weight  given  to  the  training  requirements 
index  in  the  determination  of  the  indication  for  simulation  to 
make  the  cost  savings  a  less  important  consideration  in  the 
module. 

4.  By  examining  other  tasks,  the  user  may  decide  that  the  model 
has  underestimated  the  cost  savings  that  would  actually  occur. 
In  this  case,  the  user  might  adjust  the  slope  of  the  cost 
boundary  downward  in  proportion  to  the  degree  to  which  cost 
savings  were  underestimated,  thereby  including  more  tasks  in 
the  FMS  task  cluster. 

In  this  example,  we  assume  that  the  user  makes  the  second 
judgment  listed  above.  If  the  user  increases  the  relative  weight 
for  the  operating-cost  savings  index  from  0.5  to  0.9,  the  value 
of  the  indication  for  simulation  for  most  tasks  decreases.  Four 
new  tasks  are  added  to  the  actual  equipment  task  cluster  (Figure 
6)  . 


Figure  6  is  a  representation  of  the  graphical  summary  of 
results  presented  by  OSBATS  when  the  weights  of  the  training 
requirements  index  and  operating-cost  savings  index  have  been 
adjusted.  Each  asterisk  (*)  represents  a  task.  If  two  tasks 
occupy  the  same  point  they  are  represented  by  a  number  two  (2) . 

We  assume  that  the  user  agrees  with  the  tasks  assigned  to  the  FMS 
cluster,  but  thinks  that  the  task  Simulated  Engine  Failure  might 
be  trained  better  in  an  FMS  rather  than  in  a  PMS  because  it 
requires  numerous  systems.  We  also  assume  the  user  thinks  that 
the  task  Manual  Throttle  should  remain  in  the  PMS  task  cluster. 
The  task  complexity  line  could  be  decreased  from  0.5  to  0.4  on 
the  normalized  cost  scale  to  achieve  the  desired  changes  in 
assignment  of  tasks  to  task  clusters. 

The  results  of  the  change  in  location  of  the  Task  Complexity 
Line  are  shown  in  Figure  7.  Notice  that  three  tasks.  Simulated 
Engine  Failure,  VMC  Approach  (1014)  and  Terrain  Flight  Approach 
(1022),  moved  from  the  PMS  task  cluster  to  the  FMS  task  cluster. 
The  task  Manual  Throttle  Operation  remains  in  the  PMS  task 
cluster. 

In  addition  to  changing  the  location  of  the  task  complexity 
line  and  changing  the  relative  weights  of  the  training 
requirements  index  and  the  operating-cost  savings  index,  the  user 
can  adjust  the  cost  boundary  slope.  To  assist  the  user  in 
adjusting  the  cost  boundary  slope  there  is  a  display  called 
Sensitivity  Analysis. 
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Training  Requirements  Index  =0.1  Cost  Boundary  Slope  =Base 
Operating  Cost  Savings  Index  =0.9  Task  Complexity  Line  =0.5 


Figure  6.  Graphical  summary  of  the  Simulation  Configuration 
Module  results. 


Training  Requirements  Index  =0.1  Cost  Boundary  Slope  =Base 
Operating  Cost  Savings  Index  =0.9  Task  Complexity  Line  =0.4 


Figure  7.  Graphical  summary  of  final  recommendations  of  the 
Simulation  Configuration  Module. 
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Sensitivity  analysis  and  selection  of  final  solution.  The 
final  step  in  the  analysis  is  an  examination  of  the  cost  boundary 
and  how  it  affects  the  task  clusters.  This  is  done  through  a 
sensitivity  analysis,  as  presented  in  Table  11,  in  which  the 
boundary  is  varied  from  40%  below  to  40%  above  its  nominal  value. 
A  low  slope,  which  indicates  that  the  cost  savings  has  a  higher 
weight  than  the  development  cost,  causes  more  tasks  to  be 
included  in  the  FMS  task  cluster  than  a  high  slope. 

When  the  user  has  completed  the  examination  of  the  results  of 
the  Simulation  Configuration  Module,  the  cluster  definitions  can 
be  saved  to  be  used  by  other  modules.  The  final  results  of  the 
module  are  shown  in  Table  12 .  These  clusters  form  the 
requirements  for  the  design  of  training  devices,  and  are  used  by 


Table  11 

Number  of 
is  Varied 

Tasks 

in  Each 

Cluster  as 

Cost  Boundary 

Slope 

of  Cost 

Boundary 

Task 

Cluster 

-40% 

-20% 

base 

1  +20% 

+40% 

FMS 

17 

17 

15 

13 

11 

PMS 

19 

19 

18 

18 

17 

AE 

2 

2 

5 

7 

10 

Table  12  Final  Recommendations  of  the  Simulation  Configuration 
Module  in  Tabular  Summary 


Actual  Part -Mission  Full-Mission 

Equipment  Simulator  Simulator 


Task 

#  Name 

Task 

#  Name 

Task 

#  Name 

1004 

HOVER  PWR  CK 

1003 

T/0  CHECKS 

1014 

VMC  APPROACH 

1012 

DOPPLER  NAV 

1005 

HOVER  FLT 

1015 

SHLW  APPR 

1016 

CONFIND  OPN 

1006 

NORMAL  T/0 

1019 

TERRAIN  FLT 

1042 

MOVE  TECH 

1007 

S/MAX  P  T/0 

1022 

TRN  FLT  APP 

3033 

SCAB  FAILURE 

1008 

DECEL  ACCEL 

1026 

SIM  ENG  FAIL 

1009 

TRFC  PTRN 

1048 

PIN/RDG  OPN 

1013 

PRE  LNDG  CK 

1054 

OP  RMS 

1018 

TERR  FLT  T/0 

1056 

OP  TURRET 

1021 

NOE  DECEL 

1057 

OP  FFAR  ROC 

1024 

HVR  AUTOROT 

1058 

OP  TOW  MSL 

1028 

MAN  THROTTLE 

1061 

NOE  FIRING 

1029 

NO  SCAS/SAS 

2000 

STD  AUTOROT 

1039 

MSKG/UNMSKG 

2003 

S/A  TRQ  MAL 

1055 

WPNS  PROC 

2004 

L-A,H-S  AR 

3008 

DROOP  CMP  FL 

3010 

TAIL  ROT  FL 

3011 

MAST  BUMPING 

3025 

THROTTLE  FL 

3030 

LNDG/TREES 
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the  Instructional  Feature  Selection  Module  and  the  Fidelity 
Optimization  Module.  These  two  modules  are  described  in  the 
following  sections. 

Instructional-Feature  Selection  Module 

The  Instructional-Feature  Selection  Module  begins  the 
training-device  design  process.  The  training-device  design 
process  is  conducted  for  each  training  device  incorporated  into 
the  training  system.  Any  task  cluster  may  form  the  requirements 
for  a  training  device.  The  user  may  use  any  of  the  task  clusters 
produced  by  the  Simulation  Configuration  Module  to  start  the 
training-device  design  process,  or  the  user  may  apply  the 
Instructional-Feature  Selection  Module  to  a  task  cluster  of  his 
or  her  own  making.  The  Instructional-Feature  Selection  Module 
uses  a  set  of  rules,  which  have  been  developed  through  research, 
to  compute  the  benefit  on  a  task-by-task  basis  for  each  available 
feature.  The  feature  benefits  are  compared  to  their  respective 
costs  and  a  cost-benefit  priority  list  of  features  is  developed. 
This  priority  list  can  then  be  incorporated  into  the  Fidelity 
Optimization  Modrle  so  that  the  instructional  features  can  be 
prioritized  with  training-device  fidelity  options. 

In  this  example,  we  will  evaluate  the  21  candidate 
instructional  features  shown  in  Table  13  according  to  how  well 
they  can  be  applied  in  a  training-device  to  train  the  FMS  task 
cluster  produced  by  the  Simulation  Configuration  Module.  The 
results  of  this  model  will  order  the  candidate  instructional 
features  from  most  to  least  useful  according  to  the  ratio  of 
benefit  to  development  cost.  The  user  will  then  select  several 
instructional  features  that  will  be  investigated  further  in  the 
Fidelity  Optimization  Module. 

Initial  model  solution.  The  initial  recommendations  of  the 
Instructional-Feature  Selection  Module  are  shown  in  Table  14. 

The  results  indicate  that  the  first  three  instructional  features, 
which  have  the  highest  benefit-to-cost  ratio,  offer  significant 
increase  in  training  efficiency  considering  the  cost,  the  next 
ten  features  offer  considerably  less  benefit,  and  the  eight 
remaining  instructional  features  offer  very  little  or  no  benefit. 

Figure  8  shows  graphically  the  cost  and  benefit  of  optimal 
packages  of  instructional  features.  The  first  non-zero  point  on 
the  figure  represents  the  cost  and  benefit  of  the  instructional 
feature  with  the  highest  benefit-to-cost  ratio.  In  this  example, 
the  first  point  represents  the  feature  Reset/Reposition.  The 
second  point  represents  the  cumulative  cost  and  benefit  of  the 
two  features  having  the  highest  benef it-to-rost  ratios,  and  so 
forth.  The  final  point  represents  the  cumulative  cost  and 
benefit  of  a  package  containing  all  the  instructional  features. 
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Table  13  Instructional  Features  Evaluated  by  Instructional- 
Feature  Selection  Module 


Instructional  Feature 

Abbreviation 

Automated  Performance  Measurement 

perfmeas 

Performance  Indicators 

perf  ind 

Procedure  Monitoring 

proc  mon 

Automated  Performance  Alerts 

perfalrt 

Augmented  Feedback 

aug  fdbk 

Augmented  Cues 

aug  cues 

Record/ Playback 

rec/play 

Total  System  Freeze 

sys  frez 

Remote  Graphics  Replay 

graph  rp 

Initial  Conditions 

init  end 

Scenario  Control 

seen  ctl 

Crash  Override 

crash  ov 

Reset/Reposition 

reset/re 

Parameter  Freeze 

parm  frz 

Flight  System  Freeze 

fit  frez 

Positional  Freeze 

pos  frez 

Real-Time  Simulation  Variables  Control 

realtime 

Automated  Simulator  Demonstration 

autodemo 

Adjunct  CAI 

adjn  cai 

Automated  Adaptive  Training 

adaptive 

Automated  Cueing  and  Coaching 

coaching 

The  optimal  package  of  instructional  features  can  be 
determined  from  this  graph  as  a  function  of  the  budget  for 
instructional  features.  For  example,  if  the  budget  for 
instructional  features  were  $150K  then  the  model  would  recommend 
incorporating  four  instructional  features  in  the  training  device: 
Flight  System  Freeze,  Crash  Override,  Total  System  Freeze,  and 
Reset/Reposition.  Of  course,  usually  there  will  not  be  a  budget 
designated  for  instructional  features.  In  this  case,  several 
optimal  packages  of  instructional  features  can  be  chosen  for 
evaluation  in  the  Fidelity  Optimization  Module. 

Examination  of  the  solution.  The  cost  of  each  instructional 
feature  represents  development  cost  and  is  taken  directly  from 
the  data  base.  The  benefit,  on  the  other  hand,  is  calculated 
from  the  number  of  tasks  for  which  that  instructional  feature  was 
estimated  to  improve  training  efficiency.  The  user  can  examine 
which  instructional  features  are  appropriate  for  each  task  to 
verify  the  accuracy  of  the  model  estimation  procedures.  The 
module  assigns  features  to  tasks,  as  shown  in  Table  15,  according 
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Table  14  Initial  Recommendations  of  the  Instructional-Feature 
Selection  Module 


Instructional 

Features 

Tasks 

Wts. 

Ben. 

Wt. 

Benefit  Norm. 

Ben. 

Cost 

(OOO's) 

Norm. 

Cost 

B/C 

Reset/Repos . 

89 

4.63 

4.16 

0.11 

30 

0.01 

7.39 

Total  System 

89 

3.10 

2.78 

0.07 

24 

0.01 

6.17 

Crash  Overri 

49 

4.77 

2.38 

0.06 

29 

0,01 

4.37 

Flight  Syste 

41 

3.80 

1.58 

0.04 

35 

0.02 

2.40 

Initial  Cond 

99 

4.30 

4.30 

0.11 

98 

0.05 

2.34 

Procedure  Mo 

92 

3.40 

3.16 

0.08 

74 

0.04 

2.27 

Positional  F 

41 

3.30 

1.37 

0.03 

35 

0.02 

2.08 

Parameter  Fr 

41 

2.95 

1.22 

0.03 

35 

0.02 

1.86 

Scenario  Con 

99 

4.40 

4.40 

0.11 

155 

0.07 

1.51 

Real-Time  Si 

99 

3.70 

3.70 

0.09 

137 

0.07 

1.44 

Remote  Graph 

75 

2.70 

2.03 

0.05 

76 

0.04 

1.42 

Automated  Si 

41 

2.16 

0,90 

0.02 

54 

0.03 

0.88 

Record/Playb 

89 

1.42 

1.27 

0.03 

92 

0.04 

n  n  A 

Automated  Pe 

99 

1.40 

1.40 

0.04 

215 

0.10 

0.35 

Adjunct  CAI 

89 

1.70 

1.53 

0.04 

236 

0.11 

0.34 

Automated  Cu 

89 

1.10 

0.99 

0.03 

156 

0.07 

0.34 

Automated  Ad 

99 

1.59 

1.59 

0.04 

253 

0.12 

0.33 

Augmented  Cu 

39 

1.30 

0.59 

0.01 

155 

0.07 

0.17 

Augmented  Fe 

9 

1.30 

0.13 

0.00 

97 

0.05 

0.07 

Performanc  I 

0 

1.40 

0.00 

0.00 

51 

0.02 

0.00 

Automated  Pe 

0 

1.30 

0.00 

0.00 

61 

0.03 

0.00 

to  a  set  of  rules  that  make  assignments  depending  on  basic  task 
information-processing  variables.  The  rules  are  not  accessible 
to  the  user  in  the  current  version  of  the  software.  The 
assignment  of  tasks  to  an  instructional  feature  is  illustrated  in 
Table  16. 

The  overall  instructional  feature  benefit  is  related  to  the 
number  of  tasks  for  which  the  instructional  feature  is 
appropriate.  However,  all  tasks  do  not  receive  equal  weight  in 
determining  benefit.  Tasks  that  require  more  training  receive  a 
higher  weight  in  the  benefit  calculation.  The  task  weights  are 
based  on  the  total  cost  of  training  the  tasks  to  criterion.  In 
addition,  empirical  research  has  indicated  that  some 
instructional  features  are  used  by  instructors  more  often  than 
others  even  though  they  may  be  equally  appropriate.  For  this 
reason  the  weighted  average  benefit  calculated  by  the  model  is 
moderated  by  a  feature  weight  that  depends  only  on  the 
instructional  feature.  Examination  of  the  task  weights  and 
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Line  Graph  —  Results  Scaled  in  Ascending  Order 
Feature  Wts  Included  Task  Wts  Included 
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Figure  8.  Graph  of  cost  and  benefit  of  optimal  instructional- 
feature  packages. 


feature  weights  can  give  an  indication  of  the  rationale  behind 
the  benefit  values. 

There  are  conditions  in  which  the  user  may  feel  that  it  is 
inappropriate  to  use  task  weights  or  feature  weights.  For 
example,  if  the  goal  of  a  training  device  is  to  provide 
familiarization  training  on  a  variety  of  tasks,  the  task  weights, 
which  are  based  on  training  tasks  to  criterion,  may  not  be 
appropriate.  In  this  case,  an  unweighted  average  of  tasks  may 
more  accurately  represent  the  benefit  of  an  instructional 
feature.  Similarly,  since  the  feature  weights  are  essentially  a 
measure  of  instructor  bias,  they  may  be  counteracted  by  training, 
familiarization  with  the  instructional  feature,  or  motivational 
incentives.  Thus,  the  OSBATS  user  may  feel  that  these  biases 
should  not  be  considered  in  determining  instructional-feature 
benefit. 


Table  15  Assignment  of  Instructional  Features  to  Task  1014,  VMC 
Approach 


Feature  Wts  Included 


Task  Wts  Included 


IFs  used  in  training  Task  Number  5 


Automated  Performance  Measurement  1.4 
Initial  Conditions  4.3 
Scenario  Control  4.4 
Real-Time  Simulation  Variables  Control  3.7 
Automated  Adaptive  Training  1.6 


Table  16  Assignment  of  Tasks  to  the  Instructional  Feature, 
Automated  Performance  Measurement 


Feature  Wts  Included 


Task  Wts  Included 


Tasks  and  the  Number  of  IF's  Used  in  Training 


ATM  Name  Weight 

1014  VMC  Approach  1.90 

1015  Shallow  Approach/Landing  1.90 

1019  Terrain  Flight  1.01 

1022  Terrain  Flight  Approach  0.51 

1026  Simul.  Engine  Fail.  -Altitud  0.63 

1048  Pinnacle/Ridgeline  Operation  1.80 

1054  Operate  Rocket  Mgmt.  System  9.81 

1056  Operate  197  Turret  System  7.94 

1057  Operate  2.75"  FFAR  Launcher  21.26 

1058  Operate  TOW  Missile  System  9.21 

1061  Terrain  Flight  Firing  Techs.  36.68 

2000  Standard  Autorotation  2.41 

2003  Simulated  Antitorque  Malfunc  2.41 

2004  Lo  Level  Hi  Speed  Autorotati  2.41 

3010  Tail  Rotor  Failure  at  a  Hove  0.12 


Table  17  compares  the  original  solution  with  the  solutions 
obtained  when  the  task  weights  and  the  feature  weights  are 
excluded.  Only  the  11  instructional  features  with  the  highest 
benefit-to-cost  ratios  in  each  solution  are  shown.  The  results 
are  similar  in  many  respects,  but  there  are  noticeable 
differences  between  the  solutions. 

When  the  task  weights  are  eliminated,  the  feature  named  Crash 
Override  jumps  from  third  to  first  in  the  order.  Examination  of 
which  features  are  assigned  to  each  task  gives  an  indication  as 
to  why  this  change  occurs.  The  user  could  verify 
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Table  17  Comparison  of  Solutions  when  Task  Weights  and  Feature 
Weights  are  Ignored 


Both  Weights 
Considered 

Task  Weights 
Ignored 

Feature  Weights 
Ignored 

reset/re 

crash  ov 

sys  frez 

sys  frez 

reset/re 

reset/re 

crash  ov 

sys  frez 

crash  ov 

fit  frez 

init  end 

proc  mon 

init  end 

proc  mon 

pos  frez 

proc  mon 

seen  ctl 

parm  frz 

pos  frez 

real  tim 

fit  frez 

parm  frz 

fit  frez 

init  end 

seen  ctl 

pos  frez 

graph  rp 

real  tim 

parm  frz 

rec/play 

graph  ip 

graph  rp 

autodemo 

from  the  Instructional  Feature  -  Task  Assignment  Lists  that  there 
are  10  tasks  for  which  Crash  Override  is  appropriate,  while  there 
are  8  for  Reset/Reposition  and  7  for  Total  System  Freeze. 

However,  the  tasks  assigned  to  Crash  Override  involve  a 
relatively  smaller  amount  of  training  to  reach  criterion.  Reset/ 
Reposition  and  Total  System  Freeze  are  appropriate  for  the 
difficult  weapons  tasks  that  require  much  more  training.  Given 
this  comparison,  we  will  assume  that  the  user  accepts  the 
original  solution,  because  the  FMS  would  be  expected  to  take  a 
substantial  amount  of  the  training  load  for  the  weapons  tasks. 

When  the  feature  weights  are  set  equal  to  each  other,  the 
feature  Total  System  Freeze  moves  from  second  on  the  list  to 
first.  An  examination  of  the  feature  weights  found  in  Table  14 
indicates  why  this  result  occurs.  The  feature  Reset/Reposition 
was  judged  by  instructors  to  be  very  useful,  and  consequently  has 
a  high  benefit  weight.  Total  System  Freeze,  on  the  other  hand, 
was  judged  less  likely  to  be  used,  and  received  a  weight 
approximately  two-thirds  that  of  Reset/Reposition.  In  the 
original  calculations.  Total  System  Freeze  received  a  lower 
overall  benefit  than  it  otherwise  would  have  received.  When 
feature  weights  are  excluded.  Total  System  Freeze  has  a 
relatively  higher  benefit-to-cost  ratio  than  Reset/Reposition 
because  it  is  less  expensive. 

Selecting  and  saving  instructional-feature  alternatives. 

We  assume  that  the  user  accepts  the  solution  with  both  task 
weights  and  feature  weights  included.  The  final  task  for  the 
user  is  to  choose  a  range  of  instructional  range  features  to  be 
included  in  the  Fidelity  Optimization  Module.  We  will  assume  for 
this  example  that  the  minimum  level  the  user  will  consider 
includes  the  first  three  instructional  features, 

Reset/Reposition,  Total  System  Freeze  and  Crash  Override  ($83K) . 
The  user  is  willing  to  consider  the  optional  addition  of  the 
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following  features  to  the  minimum  package.  As  a  second  level, 
the  user  may  consider  the  first  three  features  plus  Flight  System 
Freeze,  Initial  Conditions,  Procedure  Monitoring  and  Positional 
Freeze  ($325K).  The  third  level  includes  the  first  and  second 
levels  plus  Parameter  Freeze  and  Scenario  Control  ($515K) .  The 
highest  level  of  instructional  features  the  user  will  consider  is 
the  cumulative  package  including  the  first  three  packages  and 
Real-time  Simulation  Variables  Control,  Remote  Graphics  Replay, 
Automated  Simulation  Demonstration  and  Record/Playback  ($874K) . 
These  four  options  (the  minimum  option  plus  the  three  cumulative 
options)  will  then  be  incorporated  in  the  Fidelity  Optimization 
Module  to  obtain  an  overall  training-device  design. 

Fidelity  Optimization  Module 

The  Fidelity  Optimization  Module  continues  the  training- 
device  design  process  begun  with  the  Instructional  Feature 
Selection  Module.  The  goal  of  the  Fidelity  Optimization  Module 
is  to  specify  the  levels  of  technical  sophistication  on  each  of  a 
set  of  fidelity  dimensions  to  maximize  the  effectiveness  of  a 
training  device  given  its  development  cost.  The  Fidelity 
Optimization  Module  may  recommend  several  optimal  training- 
device  designs  at  different  costs.  Each  of  the  recommended 
designs  could  then  be  evaluated  by  the  Training-Device  Selection 
and  Resource  Allocation  Modules.  A  second  function  of  the 
Fidelity  Optimization  Module  is  to  make  tradeoffs  between 
technical  sophistication  in  presenting  cues  and  response  feedback 
and  technical  sophistication  in  instructional  support.  To 
accomplish  this  goal  the  Fidelity  Optimization  Module  considers 
solutions  to  the  Instructional  Feature  Selection  Module  as  an 
additional  variable  in  its  evaluation. 

In  this  example,  we  will  evaluate  eleven  candidate  fidelity 
dimensions  to  design  a  training  device  for  the  FMS  task  cluster. 
Each  of  the  fidelity  dimensions  may  be  represented  at  a  number  of 
levels  as  shown  in  Table  18.  This  model  uses  data  concerning  the 
cue  and  response  requirements  for  each  fidelity  dimension  to 
determine  the  benefit  of  each  level  of  each  dimension.  Using 
cost  formulas  based  upon  technical  performance,  cost  measures  are 
also  calculated  for  each  fidelity  option.  Then  the  fidelity 
options  across  all  fidelity  dimensions  are  ordered  according  to 
the  benefit-to-cost  ratio  and  optimal  training-device  designs  are 
defined  for  given  levels  of  development  cost.  The  user  can  then 
select  several  simulator  designs  at  desired  levels  of  cost  or 
benefit.  These  training-device  designs  can  then  be  analyzed  by 
the  Training-Device  Selection  Module. 

Initial  model  solution.  The  model  solution  is  based  on 
estimates  of  cost  and  benefit  for  the  options.  The  benefit 
values,  shown  in  Table  19,  are  based  on  the  number  of  tasks  that 
could  be  trained  at  each  level  of  each  dimension.  That  is,  for  a 
given  level  of  a  fidelity  dimension,  the  benefit  is  based  on  the 
number  of  tasks  for  which  the  technical  performance  of  that  level 
is  at  least  as  great  as  the  corresponding  task  cue  and  response 
requirement.  Tasks  are  weighted  by  the  task  weights,  as  they 
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Table  18  Dimensions  and  Levels  Considered  by  the  Fidelity 
Optimization  Module 


Dimension  1  -  Visual  Resolution 

Level  1  -  M2  at  0.3  km 
Level  2  -  M2  at  0 . 5  km 
Level  3  -  M2  at  1.0  km 
Level  4  -  M2  at  2 . 0  km 
Level  5  -  M2  at  3 . 0  km 
Level  6  -  M2  at  4.0  km 

Dimension  2  -  Visual  Content 

Level  1  -  Plane  with  Trees 

Level  2  -  Add  Generic  Features 

Level  3  -  Realistic  Density 

Level  4  -  Low  Density  Hydrographic  features 

Level  5  -  Medium  Density  Hydrographic  features 

Level  6  -  High  Density  Hydrographic  features 

Dimension  3  -  Visual  Texture 

Level  1  “  Lines  and  Polygons 
Level  2  -  Modulating  Functions 
Level  3  -  Few  Digitized  Photographs 
Level  4  -  More  Digitized  Photographs 
Level  5  -  Many  Digitized  Photographs 

Dimension  4  -  Front  Field  of  View 

Level  1  -  40  X  40  degrees 

Level  2  -  40  X  50  degrees 

Level  3  -  40  X  60  degrees 

Dimension  5  -  Side  Field  of  View 

Level  1  -  40  X  40  degrees 

Level  2  -  40  X  50  degrees 

Level  3  -  50  X  50  degrees 

Level  4  -  50  X  60  degrees 

Level  5  -  40  X  50  degrees  -  two  sides 

Level  6  -  40  X  60  degrees  -  two  sides 

Level  7  -  50  X  60  degrees  -  two  sides 


Table  18  (continued) 


Dimension  6  -  Special  Effects  -  Points 


Level  1  -  None 

Level  2  -  Cultural  Lights 

Level  3  -  Add  Weapons  Blast 

Level  4  -  Add  Damaged  Vehicles 

Level  5  -  Add  Airborn  Vehicles 

Level  6  -  Add  Moving  Ground  Vehicles 


Dimension  7  -  Special  Effects  -  Area 


Level  1  -  None 

Level  2  -  Smoke  and  Dust 

Level  3  -  Rotor  Wash 


Dimension  8  -  Motion  Platform 


Level  1  -  None 

Level  2-3  degrees  of  freedom 
Level  3-5  degrees  of  freedom 
Level  4-6  degrees  of  freedom 

Dimension  9  -  Seat  Motion 

Level  1  -  Stationary 
Level  2  -  Seat  Shaker 
Level  3  -  Add  G-Seat 

Dimension  10  -  Audio  Effects 


Level  1  -  None 

Level  2  -  Weapons,  Skid,  Failures 
Level  3  -  Add  Normal  Operating  Noise 
Level  4  -  Add  Abnormal  Operating  Noise 


Dimension  11  -  Data  Base  Size 


Level  1  - 
Level  2  - 
Level  3  - 
Level  4  - 
Level  5  - 
Level  6  - 
Level  7  - 


5  X  5  km 
10  X  10  km 
10  X  20  km 
10  X  30  km 
20  X  30  km 
30  X  30  km 
30  X  40  km 
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were  in  the  Instructional  Feature  Selection  Module.  Each 
fidelity  dimension  receives  a  weight  that  depends  on  the  overall 
pattern  of  cue  and  response  requirements  for  that  dimension.  If 
many  tasks  require  high  levels  of  technical  performance  on  a 
dimension,  then  the  dimension  will  receive  a  high  weight  in 
determining  overall  benefit  of  a  training-device  design. 

The  estimated  costs  are  shown  in  Table  20.  These  costs  are 
estimated  directly  from  technical  performance  using  an  equation 
that  assumes  that  low  levels  of  technical  performance  can  be 
obtained  relatively  cheaply  compared  to  levels  of  performance 
that  are  closer  to  the  state  of  the  art.  The  cost  estimates 
represent  development  cost  only. 

The  incremental  benefit-to-cost  ratio  for  the  fidelity- 
dimension  levels  is  shown  in  Table  21.  In  calculating  the 
incremental  benefit-to-cost  ratio,  the  module  eliminates  those 
options  that  are  not  cost-efficient  within  each  dimension.  For 
example  the  benefit-to-cost  ratio  for  the  increment  from  the 
first  to  the  second  level  of  the  dimension  Sound  Effects  is  less 
than  the  that  for  the  increment  from  the  second  level  to  the 
third.  In  this  case,  the  module  eliminates  the  second  level  from 
consideration.  It  then  recalculates  the  benefit-to-cost  ratio 
for  the  third  level  to  represent  the  increment  from  the  first 
level  to  the  third  level. 

Selection  of  candidate  training-device  designs.  The  plot  of 
the  cost  and  benefit  of  optimal  designs  is  shown  in  Figure  9. 

The  user  may  access  these  designs  by  specifying  a  target  cost. 

The  module  will  then  determine  the  design  that  has  the  greatest 
benefit  at  that  cost  or  less.  For  this  example  problem,  we 
assume  that  the  user  has  a  development  budget  of  $7  million,  but 
that  he  would  like  to  investigate  the  cost-effectiveness  of  a 
significantly  less  expensive  design  that  would  cost  near  $3 
million. 

When  the  user  specifies  the  target  cost  of  $7  million,  the 
system  specifies  the  design  shown  in  Table  22.  This  design  has 
an  estimated  development  cost  of  $5.5  million,  which  is  below  the 
budget.  The  user  might  then  examine  the  next  most  expensive 
optimal  design,  shown  in  Table  23.  This  design  is  considerably 
over  the  budget,  with  a  cost  of  $7.8  million.  Examination  of  the 
two  designs  shows  that  the  more  expensive  design  includes  a 
significantly  better  Visual  Resolution  system  than  the  less 
expensive  design.  The  cost  of  the  enhancement  is  the  difference 
between  $2391K  (Level  5)  and  $50K  (Level  1),  or  $2.3  million. 
Given  these  estimates,  the  user  would  probably  wish  to  examine 
whether  there  are  other  useful  ways  to  fill  the  $1.5  million  gap 
between  the  recommended  solution,  a  $5.5  million  design,  and  the 
$7  million  budget. 
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Table  19  Benefit  Values  of  Fidelity  Levels 


Level 

Benefit 
1  2 

3 

4 

5 

6 

7 

Visual  Resol. 

0.00 

0.40 

0.40 

1.20 

2.92 

2.92 

Visual  Content 

0.00 

3.68 

3.73 

3.73 

3.73 

3.73 

Visual  Texture 

0.36 

1.60 

1.60 

14.38 

14.38 

Visual  Front 

0.01 

9.57 

10.49 

Visual  Side 

1.86 

1.86 

2.10 

2.13 

4.88 

4.88 

4.97 

Point  Effects 

1.77 

2.37 

2.37 

2.37 

2.37 

15.70 

Area  Effects 

1.18 

7.81 

7.81 

Platform  Mot. 

0.75 

0.77 

0.81 

0.81 

Seat  Motion 

0.00 

7.57 

8.17 

Sound  Effects 

0.00 

0.00 

10.71 

11.62 

Map  Size 

0.09 

3.63 

3.63 

3.63 

3.66 

3.66 

3.66 

Inst_features 

1.69 

3.58 

4.60 

6.03 

Table  20  Cost  (in 

thousands)  of 

Fidelity  Levels 

Costs 

Level 

1 

2 

3 

4 

5 

6 

7 

Visual  Resol. 

50 

887 

1744 

2235 

2391 

2471 

Visual  Content 

80 

496 

942 

1424 

1947 

2520 

Visual  Texture 

75 

393 

758 

1108 

1344 

Visual  Front 

20 

403 

1103 

Visual  Side 

10 

43 

85 

121 

194 

248 

319 

Point  Effects 

0 

38 

79 

167 

318 

504 

Area  Effects 

0 

147 

336 

Platform  Mot. 

0 

1173 

2084 

3024 

Seat  Motion 

0 

67 

192 

Sound  Effects 

0 

43 

96 

192 

Map  Size 

100 

634 

918 

1216 

1855 

2564 

3360 

Inst_features 

83 

325 

515 

874 
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Table  21  Incremental  Benef it-to-Cost  Ratios  for  Fidelity  Levels 


Level 

1 

2 

Benefit/Cost 
3  4 

Ratio 

5  6  7 

Visual  Resol. 

undef 

0.12  - 

Visual  Content 

undef 

0.88 

0.01 

— 

-  - 

Visual  Texture 

unde;  ; 

— 

— 

1.36 

— 

Visual  Front 

undef 

2.49 

0.13 

Visual  Side 

undef 

— 

— 

— 

1.64  -  0.07 

Point  Effects 

undef 

— 

— 

— 

-  2.76 

Area  Effects 

undef 

4.51 

— 

Platform  Mot. 

undef 

— 

0.00 

— 

Seat  Motion 

undef 

11.16 

0.48 

Sound  Effects 

undef 

— 

11.13 

0.95 

Map  Size 

undef 

0.66 

— 

— 

0.00  -  - 

Inst  features 

undef 

0.78 

0.54 

0.40 

Table  22  Optimal 
Million 

Training-Device  Design  at 

a  Cost  of  $5.5 

Dimension 

Level 

Description 

Benefit 

Cost 

Visual  Resol. 

1 

m2  at  0.3  km 

0.0 

50.00 

Visual  Content 

2 

Add  Generic  Featrs 

3.7 

496.08 

Visual  Texture 

4 

More  Digit  Ph 

14.4 

1108.32 

Visual  Front 

3 

40  X  60  Deg 

10.5 

1103.93 

Visual  Side 

5 

40  X  50  Deg 

4.9 

194.67 

Point  Effects 

6 

Add  Mvng  Grnd  Veh 

15.7 

504.00 

Area  Effects 

2 

smoke  and  dust 

7.8 

147.22 

Platform.  Mot. 

1 

none 

0.8 

0.00 

Seat  Motion 

3 

Add  G-Seat 

8.2 

192.00 

Sound  Effects 

4 

Add  abnor  opt  nse 

11.6 

192 . 00 

Map  Size 

2 

10  X  10  km 

3.6 

634.40 

inst_features 

4 

Inst  Feat,  Lev  4 

6.0 

874.00 

Totals 

87.1 

5496.62 
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Figure  9.  Graph  of  cost  and  benefit  of  optimal  training-device 
designs:  Initial  analysis. 

Table  23  Optimal  Training-Device  Design  at  a  Cost  of  $7.8 
Million 


Dimension 

Level 

Description 

Benefit 

Cost 

Visual  Resol. 

5 

m2  at  3.0  km 

2.9 

2391.12 

Visual  Content 

2 

Add  Genric  Featrs 

3.7 

496.08 

Visual  Texture 

4 

More  Digit  Ph 

14.4 

1108.32 

Visual  Front 

3 

40  X  60  Deg 

10.5 

1103 . 93 

Visual  Side 

5 

40  X  50  Deg 

4.9 

194 . 67 

Point  Effects 

6 

Add  Mvng  Grnd  Veh 

15.7 

504.00 

Area  Effects 

2 

smoke  and  dust 

7.8 

147.22 

Platform  Mot. 

1 

none 

0.8 

0.00 

Seat  Motion 

3 

Add  G-Seat 

8.2 

192 . 00 

Sound  Effects 

4 

Add  abnor  opt  nse 

11.6 

192.00 

Map  Size 

2 

10  X  10  km 

3.6 

634.40 

inst_features 

4 

Inst  Feat,  Lev  4 

6.0 

874 . 00 

Totals 

90.1 

7837.74 
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This  problem  may  be  addressed  by  eliminating  the  enhanced 
visual  resolution  system  from  consideration  and  recalculating  the 
optimal  designs.  The  training-device  designs  will  be 
recalculated  and  the  $1.5  million  formerly  designated  for  visual 
resolution  can  be  allocated  to  other  options.  The  results  show 
that  this  gap  is  reduced.  The  optimal  design  shown  in  Table  24 
is  found  with  a  cost  of  $6.9  million,  which  is  closer  to  the 
budget.  Note  that  the  recommended  design  now  includes  the  second 
level  of  visual  resolution,  which  costs  an  additional  $837,000 
over  the  first  level.  It  also  includes  an  additional  level  of 
the  dimension  Visual  Content,  at  the  additional  cost  of  $450,000, 
and  two  additional  levels  of  the  dimension  Side  Field-of-View, 
for  $125,000.  The  total  additional  cost  over  the  $5.5  million 
dollar  design  is  $1.4  million. 

We  assume  the  user  is  satisfied  with  this  design  and  wishes  to 
save  it  for  further  evaluation  in  the  Training-Device  Selection 
module.  In  addition,  the  design  shown  in  Table  25  at  a  cost  of 
$2.9  million  is  selected.  These  two  designs  are  then  saved  to  be 
used  as  the  basis  of  the  analyses  performed  by  the  Training- 
Device  Selection  Module. 

Table  24  Optimal  Training-Device  Design  at  a  Cost  of  $6.9 
Million 


Dimension 

Level 

Description 

Benefit 

Cost 

Visual  Resol. 

2 

m2  at  0.5  km 

0.4 

887.71 

Visual  Content 

3 

Realistic  Densty 

3.7 

942 . 64 

Visual  Texture 

4 

More  Digit  Ph 

14 . 4 

1108 . 32 

Visual  Front 

3 

40  X  60  Deg 

10.5 

1103.93 

Visual  Side 

7 

50  X  60  Deg 

5.0 

319.68 

Point  Effects 

6 

Add  Mvng  Grnd  Veh 

15.7 

504 . 00 

Area  Effects 

2 

smoke  and  dust 

7.8 

147.22 

Platform  Mot. 

1 

none 

0.8 

0.00 

Seat  Motion 

3 

Add  G-Seat 

8 . 2 

192 . 00 

Sound  Effects 

4 

Add  abnor  opt  nse 

11.6 

192 . 00 

Map  Size 

2 

10  X  10  km 

3.6 

634.40 

Inst_features 

4 

Inst  Feat,  Lev  4 

6.0 

874 . 00 

Totals 

87.7 

6905.89 

260 


Table  25  Optimal  Training-Device  Design  at  a  Cost  of  $2.9 
Million 


Dimension 

Level 

Description 

Benefit 

Cost 

Visual  Resol. 

1 

m2  at  0.3  km 

0.0 

50.00 

Visual  Content 

1 

Plane  w/  Trees 

0.0 

80.00 

Visual  Texture 

4 

More  Digit  Ph 

14.4 

1108.32 

Visual  Front 

2 

40  X  50  Deg 

9.6 

403.27 

Visual  Side 

5 

40  X  50  Deg 

4.9 

194.67 

Point  Effects 

6 

Add  Mvng  Grnd  Veh 

15.7 

.  504 . 00 

Area  Effects 

2 

smoke  and  dust 

7.8 

147.22 

Platform  Mot. 

1 

none 

0.8 

0.00 

Seat  Motion 

2 

Seat  Shaker 

7.6 

67.88 

Sound  Effects 

4 

Add  abnor  opt  nse 

11.6 

192 . 00 

Map  Size 

1 

5  X  5  km 

0.1 

100.00 

Inst_f eatures 

1 

Inst  Feat,  Lev  1 

1.7 

83 . 00 

Totals 

74 . 1 

2930.36 

Training-Device  Selection  Module 

The  Training-Device  Selection  Module  takes  as  input  a  set  of 
candidate  training  devices  for  the  set  of  tasks.  These  training 
devices  include  full-mission  simulators,  part-mission  simulators, 
and  other  training  devices  designed  by  the  Instructional  Feature 
Selection  and  Fidelity  Optimization  Modules.  Also  included  are 
definitions  of  existing  training  devices  or  templates  describing 
training-device  prototypes.  Unless  the  user  is  considering  the 
purchase  of  two  different  full-mission  simulators,  this  module 
should  be  exercised  separately  for  each  FMS  that  the  user  defined 
in  the  Fidelity  Optimization  Module.  The  Training-Device 
Selection  Module  then  minimizes  the  cost  to  train  each  task  with 
the  devices  available,  using  learning  curve  data  for  each  task 
and  training-device  combination.  The  output  of  this  model  is  the 
aggregated  usage  of  each  training  device  across  all  tasks.  This 
module  does  not  consider  complex  cost  functions  or  task 
interactions  in  determining  the  optimal  assignment  of  training 
devices  to  tasks;  this  more  complex  task  is  performed  by  the 
Resource  Allocation  Module. 

In  this  example,  we  will  compare  the  two  FMS  candidates 
developed  in  the  Fidelity  Optimization  Module  in  a  training 
system  that  consists  of  actual  equipment,  classroom,  and  other 
training  devices.  The  complete  list  of  available  training 
devices  and  their  abbreviated  names  is  shown  in  Table  26.  For 
simplicity,  the  example  analyses  will  focus  on  small  subsets  of 
these  training  devices.  For  each  subset  of  training  devices,  the 
module  calculates  the  minimum  cost  to  meet  the  training 
requirements  using  the  specified  training  devices.  The  user  may 
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then  examine  which  tasks  are  assigned  to  which  device  and  the 
extent  to  which  each  device  is  used  to  modify  the  subset  of 
training  devices  being  considered. 

The  model  allocates  tasks  to  training  devices  by  choosing  «t 
each  level  of  performance  the  training  device  that  produces  the 
greatest  improvement  in  performance  for  the  cost.  Training 
continues  on  this  device  until  a  performance  level  is  reached  at 
which  the  training  device  is  no  longer  optimal.  Training  is  then 
switched  to  the  device  that  is  optimal  at  that  performance  level. 
This  process  continues  until  the  training  standard  is  reached. 
When  training  devices  have  been  assigned  to  all  tasks,  a  summary 
of  training-device  use  is  calculated  and  presented  to  the  user. 

To  provide  a  baseline  measure,  we  will  first  apply  the 
Training-Device  Selection  Module  to  a  training  system  consisting 
of  classroom  training  and  actual  equipment  only.  The  model 
estimates  the  per-student  cost  of  this  training  system  as  $51,900 
and  the  per-student  training  time  as  65.8  hours.  The  cost  value 
will  be  used  to  compare  the  savings  that  would  occur  with  the 
introduction  of  additional  training  devices  into  the  system. 

Table  26  List  of  Training  Devices  Evaluated  by  the  Training- 
Device  Selection  Module 


Abbreviation  Training-Device  Name 


CLASSROOM 

CPT 

DOPPLER 
IMC  FT 
VMC  FT 
WEAPON 
NOE  NAV 
ACT  EQUIP 
sevenmillion 
threemillion 


Classroom  Training 

Cockpit  Procedures  Trainer 
Doppler  Navigation  Trainer 
Instrument  Conditions  Flight  Trainer 
Visual  Conditions  Flight  Trainer 
Weapon  System  Procedures  Trainer 
Nap-of-the-Earth  Navigation  Trainer 
Actual  Equipment 

$7  million  FMS  from  Fidelity  Optimization 
$3  million  FMS  from  Fidelity  Optimization 


Analysis  of  the  $7  million  FMS.  We  now  introduce  the  $7 
million  FMS  into  the  system  and  examine  the  solution  presented  in 
Table  27.  These  results  show  an  overall  per-student  cost  of 
$40,500,  more  than  a  ten-thousand  dollar  improvement  over  the 
baseline.  The  $7  million  FMS  takes  over  a  large  amount  of  the 
training  that  was  conducted  on  the  actual  equipment  in  the 
baseline  calculations.  Though  the  total  hours  of  training  per 
student  (92.8  hours)  have  increased  compared  to  the  baseline,  the 
total  cost  per  student  has  decreased. 

Analysis  of  the  $3  million  FMS.  In  the  second  part  of  this 
analysis  the  $3  million  FMS  is  included  in  the  training  system  in 
place  of  the  $7  million  FMS.  The  results,  shown  in  Table  28, 
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indicate  an  overall  per-student  cost  of  $37,900,  a  savings  of 
$2,600  per  student  compared  to  the  first  training  system.  Though 
this  second  training  system  involves  more  training  hours  per 
student  (128  hrs.),  the  overall  cost  is  lov'er  than  the  svstem 
which  includes  the  $7  million  FMS.  Therefore,  the  $3  million  FMS 
is  superior  to  the  $7  million  FMS.  The  user  has  access  to 
information  about  the  training  devices  selected  for  each  task. 

The  $3  million  FMS  is  used  to  train  more  tasks  than  it  was 
designed  for.  Of  the  15  tasks  in  the  FMS  task  cluster  12  are 
assigned  training  on  the  FMS.  The  other  three  are  trained 
entirely  on  the  actual  equipment.  In  addition,  13  of  the  18 
tasks  in  the  PMS  task  cluster  are  trained  partially  or  fully  on 
the  FMS.  This  result  makes  sense  since  we  did  not  design  a 
training  device  for  the  task  in  the  PMS  cluster.  However,  the 
FMS  might  have  greater  fidelity  than  is  actually  required  since 
its  design  considered  the  requirements  of  three  tasks  for  which 
it  was  not  used.  Consequently,  the  results  of  the  Training- 
Device  Selection  Module  have  implications  for  earlier  modules. 

Table  27  Training-Device  Selection  Module  Results  for  the  $7 
Million  FMS 


Training  Tasks 

Devices  Trained 

of  38 

Cost  to 
Train 
(OOO's) 

Training 
Hours  / 
Student 

Dev. 

Rqd. 

Hours  / 
Year  / 
Device 

Assumed 
Hours  / 
Dev/Yr 

Classroom 

4 

0.19 

5.97 

3 

1593 

2000 

Training 

Actual 

20 

19.46 

22.49 

8 

2249 

2500 

Equipment 

sevenmillion 

31 

20.87 

64.38 

18 

2860 

3000 

Totals 

40.52 

92.84 

Table  28 

Training-Device  Selection  Module  Results  for 
Million  FMS 

the  $3 

Training 

Devices 

Tasks 
Trained 
of  38 

Cost  to 
Train 
(000s) 

Training 
Hours  / 
Student 

Dev. 

Rqd. 

Hours  / 
Year  / 
Device 

Assumed 
Hours  / 
Dev/Yr 

Classroom 

4 

0.19 

5.97 

3 

1593 

2000 

Training 

Actual 

29 

24.82 

28.69 

10 

2295 

2500 

Equipment 

threemillion  27 

12.84 

93.31 

25 

2986 

3000 

Totals 

37 . 85 

127.98 
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To  further  investigate  this  aspect  of  the  results,  the  user 
would  define  a  smaller  FMS  task  cluster  to  use  as  the  basis  for 
the  Instructional  Feature  Selection  and  Fidelity  Optimization 
Modules.  For  example,  the  user  might  select  to  include  in  the 
FMS  task  cluster  only  those  12  tasks  that  were  assigned  training 
on  the  FMS.  This  would  result  in  a  smaller  FMS  task  cluster. 

The  training-device  designs  produced  by  the  Instructional 
Features  and  Fidelity  Optimization  modules  would  be  specific  for 
a  smaller  number  of  tasks  and  therefore  would  probably  have  lower 
fidelity  requirements.  These  designs  would  then  be  evaluated  by 
the  Training-Device  Selection  Module  to  determine  if  further 
savings  could  be  obtained. 

The  user  may  wish  to  compare  the  results  obtained  using  the  $3 
million  FMS  with  the  results  obtained  from  using  one  of  the 
prototype  training  devices.  There  are  a  number  of  weapons  tasks 
for  which  the  students  have  very  low  entry  levels  of  knowledge. 

It  is  possible  that  the  training  system  would  be  less  expensive 
if  a  weapons  trainer  were  used.  Since  a  template  for  a  weapons 
system  trainer  exists  in  the  data  base,  we  will  add  it  to  the 
list  of  devices  to  replace  the  current  FMS.  The  results  shown  in 
Table  29  indicate  that  the  total  per-student  cost  for  this 
training  system  is  $49,000,  more  than  ten-thousand  dollars  over 
the  training  system  with  the  $3  million  FMS. 


Table  29  Training-Device  Selection  Module  Results  for  the  Weapon 
System  Trainer 


Training  Tasks 

Cost  to 

Training 

Dev. 

Hours  / 

Assumed 

Devices  Trained 

Train 

Hours  / 

Rqd. 

Year  / 

Hours  / 

of  38 

(000s) 

Student 

Device 

Dev/Yr 

Classroom  4 

0.19 

5.97 

3 

1593 

2000 

Training 

Weapons  System  8 

2.79 

32.39 

13 

1993 

2000 

Trainer 

Actual  33 

46.31 

53.54 

18 

2379 

2500 

Equipment 

Totals 

49.29 

91.90 

Re-iteration  of  the  Results.  The  calculations  for  the  per- 
student  cost  and  per-student  time  to  train  the  tasks  depend  on  an 
estimate  of  the  annual  usage  of  each  device.  Sometimes  the  value 
of  the  calculated  annual  usage  is  very  different  from  the 
estimated  value. 

The  Re-iteration  function  is  used  when  there  is  a  large 
discrepancy  between  the  "Assumed  Hours/Device/Year"  and  the 
actual  "Hours/Device/Year."  The  value  of  the  actual  annual  usage 
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is  substituted  for  the  estimated  annual  usage  and  the  module 
recalculates  the  results.  If  the  discrepancy  is  small  it  is  not 
necessary  to  use  Re-iteration. 

Recall  that  the  value  of  the  actual  hours  used  /  year  /  device 
for  the  $3  million  FMS  (2986  hours)  was  very  close  to  the  value 
that  was  used  to  estimate  the  hourly  cost  (3000  hours) .  In  this 
case  Re-iteration  is  not  necessary. 

Conclusions  of  the  Analysis.  It  is  clear  from  the  analysis 
that  the  $3  million  FMS  is  superior  to  the  $7  million  FMS.  The 
cost  of  the  $7  million  FMS  was  too  high  for  it  to  replace 
significant  amounts  of  actual  equipment  training  at  a  cost 
comparable  to  the  cost  of  the  $3  million  FMS.  For  the  purposes 
of  this  example  we  assume  that  the  training  system  that  will  be 
analyzed  in  further  detail  in  the  Resource  Allocation  Module 
includes  the  classroom  trainer,  the  $3  million  FMS,  and  the 
actual  equipment. 

The  results  of  the  Training-Device  Selection  Module  have 
several  implications  for  reanalysis  of  the  problem  by  earlier 
modules.  The  implications  are  enumerated  below. 

1.  The  Instructional  Feature  Selection  and  Fidelity  Optimization 
Modules  might  be  run  using  a  reduced  task  set.  The  set  might 
include  tasks  in  the  FMS  task  cluster  for  which  training  on 
the  FMS  was  selected  by  the  Training-  Device  Selection  Module. 
You  may  also  wish  to  include  those  tasks  from  the  PMS  task 
cluster  for  which  a  substantial  amount  of  training  was 
selected  to  be  on  the  FMS  by  the  Training-Device  Selection 
Module. 

2.  The  user  might  want  to  design  a  device  based  on  a  task  cluster 
consisting  of  only  the  three  tasks  from  the  FMS  task  cluster 
for  which  training  on  the  actual  equipment  was  selected  by  the 
Training-Device  Selection  Module.  The  three  tasks  are  Terrain 
Flight,  Pinnacle  Ridgeline  Operation  and  Standard 
Autorotation.  The  user  might  decide  to  rerun  Training-Device 
Selection  and  include  this  design  in  the  current  training 
system  to  see  if  the  total  cost  to  train  each  student 
decreases  even  further. 

3.  The  current  analysis  of  the  Fidelity  Optimization  Module  could 
be  examined  to  identify  other  lower-fidelity  options  that 
might  be  more  cost-efficient.  Since  the  $3  million  solution 
performed  better  than  the  $7  million  solution,  there  may  be 
other  candidate  designs  less  expensive  than  the  $3  million 
simulator  that  would  perform  even  better. 

Resource  Allocation  Module 


The  Resource  Allocation  Module  provides  results  that  are 
similar  to  those  provided  by  the  Training-Device  Selection 
Module.  However,  these  methods  differ  from  those  of  the 
Training-Device  Selection  Module  in  two  important  respects.  (a) 
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The  Training-Device  Selection  Module  assumes  that  cost  is  a 
simple  linear  function  of  training-device  use,  while  the  Resource 
Allocation  Module  breaks  the  cost  curve  into  a  number  of  linear 
segments.  (b)  The  Resource  Allocation  Module  allows  the  user  to 
specify  constraints  on  training-device  use.  These  constraints 
specify  the  maximum  time  that  a  training  device  may  be  used  or 
the  minimum  performance  level  for  which  a  training  device  may  be 
employed  on  a  specific  task. 

As  a  result  of  its  increased  accuracy  and  flexibility,  the 
Resource  Allocation  Module  involves  greater  computational 
complexity  and  calculation  time  than  the  Training-Device 
Selection  Module.  For  the  current  example  problem,  the 
calculations  of  the  Resource  Allocation  Module  can  take  from  two 
to  twenty  minutes,  depending  on  the  complexity  of  the  problem. 
This  increased  computational  load  limits  the  extent  to  which  the 
Resource  Allocation  Module  may  be  used  interactively,  although 
limited  interactions  are  possible. 

In  the  example  problem,  we  apply  the  Resource  Allocation 
Module  twice  to  the  results  of  the  Training-Device  Selection 
Module.  The  first  time,  we  will  not  apply  any  external 
constraints  to  the  solution.  The  second  time  will  add 
constraints  and  determine  their  effect  on  the  solution. 

Unconstrained  solution.  Table  30  shows  a  solution  to  the  same 
problem  that  was  addressed  in  the  Training-Device  Selection 
Module  (see  Table  28) .  Differences  in  the  solution  come  from  the 
increased  accuracy  of  the  cost  function.  Thus,  the  Resource 
Allocation  will  tend  to  allocate  time  to  a  training  device  until 
the  device  is  fully  utilized.  Then  it  will  not  allocate 
additional  time  unless  there  is  sufficient  time  to  justify  the 
purchase  of  an  additional  device. 

Constrained  solution.  In  this  analysis,  we  constrain  the 
hours  in  which  the  $3  million  FMS  may  be  used.  The  initial 
results  suggest  that  27  such  training  devices  would  be  required. 
If  we  assume  that  only  12  $3  million  FMSs  are  available,  then 
each  student  can  be  allocated  a  maximum  of  45.00  hours  on  the 
FMS. 

The  results  of  this  analysis,  shown  in  Table  31,  indicate  that 
the  training  requirements  can  be  met  under  the  new  constraints, 
although  at  a  somewhat  greater  cost.  The  increased  per-student 
cost  is  the  direct  result  of  the  addition  of  the  constraint.  The 
mandated  reduction  in  the  $3  million  FMS  was 

compensated  for  by  increased  use  of  the  actual  equipment,  which 
is  more  expensive  than  the  $3  million  FMS. 

Summary  of  Example  Problem 

The  sample  problem  has  illustrated  how  the  OSBATS  modules 
interact  to  make  recommendations  about  the  kinds  of  training 
devices  that  should  be  developed,  the  specific  device  designs 
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Table  30  Unconstrained  Solution  to  Resource  Allocation  Problem 


Training  Devices 

Tasks 
Trained 
of  38 

Cost  to 
Train 
(OOO's) 

Training 

Hours 

Student 

Dev. 

Rqd. 

Hours 
/  Year 

Classroom  Training 

4 

0.20 

5.94 

3 

4752 

Actual  Equipment 

29 

24.13 

27.80 

9 

22241 

threemillion 

27 

13.87 

99.84 

27 

79870 

Totals 

38.20 

133.58 

Table  31  Constrained  Solution  to  the 

Resource 

Allocation  Problem 

Training  Devices 

Tasks 
Trained 
of  38 

Cost  to 
Train 
(OOO's) 

Training 

Hours 

Student 

Dev. 

Rqd. 

Hours 
/  Year 

Classroom  Training 

4 

0.20 

5.94 

3 

4752 

Actual  Equipment 

33 

34.13 

39.00 

13 

31202 

threemillion 

20 

6.19 

44.88 

12 

35903 

Totals 

40.51 

89.82 

that  should  be  employed,  and  the  extent  to  which  trainin'^  devices 
should  be  used.  The  five  modules  performed  the  following 
functions. 

1.  The  Simulation  Configuration  Module  organized  the  training 
requirements  to  specify  the  relative  need  for  training  in  a 
simulated  environment.  In  addition,  the  module  recommended 
which  tasks  would  best  be  trained  by  a  full-mission  simulator 
rather  than  a  simpler  training  device. 

2.  The  Instructional  Feature  Selection  Module  evaluated  a  range 
of  instructional  support  features  for  a  full-mission 
simulator,  and  specified  the  optimal  set  of  features  to 
include  at  several  budget  levels. 

3.  The  Fidelity  Optimization  Module  evaluated  several  critical 
areas  in  which  the  technical  capability  of  a  training  device 
to  present  cues  and  response  feedback  could  vary,  and 
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recommended  the  optimal  levels  of  these  areas  as  a  function  of 
the  training-device  investment  budget. 

4.  The  Training-Device  Selection  Module  evaluated  the  training- 
device  designs  produced  by  the  Fidelity  Optimization  Module  in 
the  context  of  the  entire  set  of  training  requirements.  It 
evaluated  the  FMS  alternatives  and  specified  which  tasks  would 
use  each  available  training  device  to  meet  training 
requirements  at  the  minimum  cost. 

5.  The  Resource  Allocation  Module  refined  the  solution  of  the 
Training-Device  Selection  Module  to  account  for  complexities 
in  the  cost  function  and  constraints  in  training  device 
availability  and  use. 

This  analysis  is  illustrative  and  is  not  complete.  The 
description  notes  several  places  where  further  analysis  would  be 
required  to  arrive  at  a  definitive  recommendation.  However,  the 
example  points  out  how  the  OSBATS  model  can  be  applied  to 
concept-formulation  problems  of  reasonable  complexity. 
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APPLICATION  OF  OSBATS  TO  THE 
Ml  ABRAMS  TANK  TURRET  MECHANICS  COURSE 


The  OSBATS  model  was  designed  to  be  a  general  tool  to  aid  the 
training  device  concept  formulation  process.  However,  its  only 
application  has  been  to  a  sample  problem  from  the  domain  of 
aviation.  Application  to  other  areas  may  require  changes  to  data 
values,  model  parameters,  variables,  fidelity  dimensions  and 
levels,  instructional  features,  or  even  to  the  overall  model 
structure.  This  section  investigates  the  nature  and  extent  of 
changes  required  to  apply  the  OSBATS  model  to  a  different  domain, 
specifically  that  of  armor  turret  maintenance. 

The  specific  domain  that  will  be  analyzed  in  this  section  is 
the  Ml  Abrams  Tank  Turret  Mechanic  course  (45E10) .  The  proponent 
for  this  course  is  the  U.S.  Army  Ordnance  Center  and  School  at 
Aberdeen  Proving  Ground,  Maryland.  However,  the  course  is  taught 
at  the  U.S.  Army  Armor  School  at  Ft.  Knox,  Kentucky.  This  course 
differs  in  many  important  respects  from  the  AH-1  Airman 
Qualification  Course  (AQC)  that  formed  the  basis  of  the  initial 
application  of  the  OSBATS  model. 

1.  The  turret  mechanic  course  involves  armor,  while  the  AH-1  AQC 
involves  aviation. 

2.  The  turret  mechanic  course  involves  maintenance,  while  the  AH- 
1  AQC  involves  operation. 

3.  The  turret  mechanic  course  involves  initial  acquisition  of 
skills,  while  the  AH-1  AQC  involves  transition  of  skills  to  a 
new  weapon  system. 

4.  The  turret  mechanic  tasks  are  heavily  loaded  on  procedural  and 
cognitive  activities,  while  the  AH-1  tasks  are  heavily  loaded 
on  psychomotor  activities. 

5.  The  turret  mechanic  course  is  taught  to  enlisted  personnel 
immediately  following  basic  training,  while  the  AH-1  AQC  is 
taught  to  officers  and  warrant  officers. 

The  great  differences  between  these  two  courses  provides  a 
good  test  for  the  generality  of  the  OSBATS  model.  In  our 
analysis  of  the  turret  mechanic  course,  we  will  concentrate  on 
two  questions.  The  first  question  is  whether  the  OSBATS  process 
will  work  in  a  training  domain  that  is  considerably  different 
from  its  original  application.  The  second  question  is  what 
changes  will  be  required  to  the  data  and  procedures  for 
successful  application  of  the  OSBATS  model  in  the  new  situation. 
Our  intuitions  regarding  these  questions  are  that  the  model  will 
stand  unchanged,  but  will  have  domain-specific  data  requirements. 

One  focus  of  our  comparison  of  the  two  training  domains  will 
be  on  whether  we  can  develop  general  procedures  to  specify  the 
appropriate  data  variables,  such  as  fidelity  dimensions  and 
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instructional  features;  determine  measurement  scales;  and  assess 
data  values  for  a  new  training  domain.  The  extent  to  which  we 
can  develop  these  procedures  will  have  a  great  impact  on  the 
operating  procedures  used  by  the  OSBATS  model  and  will  to  a  great 
extent  determine  its  general  applicability. 

In  the  remainder  of  this  section,  we  will  describe  the  Ml  tank 
turret  mechanic  course,  the  data  required  to  analyze  this  course 
using  the  OSBATS  model,  and  the  adequacy  of  the  decision  process 
specified  by  the  OSBATS  model  for  providing  training  device 
design  guidance.  First,  we  will  give  a  general  description  of 
the  training  domain,  including  the  tasks  to  be  trained  and  the 
existing  training  devices.  Then,  we  will  describe  the 
requirements  for  both  resident  and  task  data.  Finally,  we  will 
examine  the  model  process  for  the  turret  mechanic  course,  and 
summarize  the  differences  between  the  two  applications  of  the 
model . 


General  Description  of  Training  Domain 

The  training  domain  selected  for  this  second  application  of 
OSBATS  is  defined  by  those  tasks  currently  listed  in  the  Program 
of  Instruction  (POI)  for  the  Ml  Abrams  Tank  Turret  Mechanic 
Course  (643-45E10) .  The  purpose  of  this  entry-level  course  is  to 
"...train  enlisted  personnel  to  perform  unit  maintenance  on  the 
vehicle-mounted  armament,  associated  fire-control,  and  related 
systems  on  the  Ml/MlAl  tank  turret  system"  (POI  643-45E10,  p.  3) . 
The  course  provides  training  on  61  of  the  122  Skill  Level  One 
tasks  described  in  the  Soldier's  Manual  for  MOS  45E  (STP  9- 
45E12-SM-TG) .  The  61  tasks  in  the  present  POI  comprise  both  duty 
position  and  related  technical  tasks.  As  described  in  the 
Soldiers  Manual,  duty  position  tasks  are  those  that  the  student 
must  learn  thoroughly,  because  he  is  expected  to  perform  them  on 
a  daily  basis.  One  or  more  related  technical  tasks  are  listed 
with  most  duty  position  tasks.  It  is  assumed  that  if  the  student 
can  perform  the  duty  position  task,  he  should  also  be  able  to 
perform  any  related  technical  task.  Because  of  this  distinction, 
the  Soldier's  Manual  provides  more  documentation  for  duty 
position  as  opposed  to  related  technical  tasks. 

Tasks 


The  present  section  describes  turret  mechanic  tasks  in  terms 
of  two  general  classification  schemes.  The  section  concludes 
with  a  description  of  how  these  classification  schemes  were  used 
to  select  a  subset  of  tasks  for  further  study. 

Functional  systems.  Both  the  Ml  Organizational  Maintenance 
Manual  (TM  9-2350-255-20-2-3-1/2/3)  and  the  Organizational 
Troubleshooting  Manual  (TM  9-2350-255-20-2-2-1/2/3)  classify 
maintenance  tasks  according  to  the  three  major  functional  systems 
of  the  Ml  turret:  (a)  the  hydraulic  system,  (b)  the  fire  control 
system,  and  (c)  the  electrical  system.  The  POI  provides  a 
similar  categorization.  Each  of  the  61  POI  tasks  selected  for 
institutional  training  is  assigned  to  a  block  of  instruction. 
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The  blocks  of  instruction  are,  in  turn,  clustered  into  eight 
annexes  or  general  topics  for  instruction.  The  three  annexes 
that  provide  most  of  the  course  content  correspond  to  these  three 
functional  systems.  Thus,  this  functional  classification  of 
tasks  appears  to  be  a  well  accepted  and  useful  taxonomy. 

Skill  requirements.  Alternatively,  turret  maintenance  tasks 
can  be  classified  according  to  their  human  skill  requirements. 
Examination  of  task  titles  revealed  four  homogeneous  categories 
of  tasks.  The  following  categories  are  similar  to  some  of  those 
described  by  Aagard  and  Braby  (1976). 

1.  Perform  Procedures.  Procedural  maintenance  tasks  are  those 
whose  titles  use  action  verbs  such  as  "remove,"  "install," 
"replace,"  and  "bleed."  These  tasks  require  the  mechanic  to 
perform  a  set  sequence  of  discrete  actions  with  to  accomplish 
some  tangible  result.  Some  of  the  actions  require  some  gross 
motor  skills,  but  the  emphasis  in  this  category  is  on 
remembering  and  performing  the  actions  in  the  correct 
sequence. 

2.  Detect/Classify.  This  category  comprises  "check,"  "adjust," 
"service,"  "inspect,"  and  "maintain"  tasks.  These  tasks  are 
very  similar  to  the  previous  category  in  that  they  require  the 
mechanic  to  perform  discrete  actions  in  a  prescribed  sequence. 
The  unique  aspect  of  these  tasks  is  their  intangible  purpose, 
i.e.,  to  detect  and  classify  mechanical  faults. 

3.  Troubleshoot.  Troubleshooting  tasks  require  the  mechanic  to 
detect,  identify,  and  correct  faults  in  any  one  of  the  three 
major  subsystems.  The  identification  of  these  faults  requires 
the  subject  to  perform  some  of  the  detect/classify  tasks  as 
described  in  the  previous  category,  while  the  prescribed 
correction  may  entail  certain  procedures  as  described  in  the 
first  category.  The  distinction  between  this  category  and  the 
previous  two  is  that  the  task  steps  are  not  performed  in  a  set 
sequence;  rather,  the  task  sequence  follows  complex  flow 
diagrams  as  presented  in  the  troubleshooting  manual.  Thus, 
troubleshooting  requires  both  rule  using  and  decision  making 
skills,  which  are  based  on  the  student's  general  knowledge  of 
electronics  and  hydraulics  and  specific  verbal  information 
related  to  repairing  the  turret  components  of  the  Ml.  In 
addition,  using  the  STE-Ml  equipment  and  schematic  diagrams 
requires  that  the  student  be  able  to  recognize  and  identify 
symbolic  information. 

4.  Prepare  forms.  Tasks  in  this  category  are  unlike  those  in 
previous  categories  in  that  they  do  not  have  either  a 
procedural  or  gross  motor  skill  component.  On  the  other  hand, 
forms  are  prepared  according  to  specific  rules  based  on 
specific  verbal  information  either  presented  on  the  form  or  in 
the  block  of  instruction  on  maintenance  forms.  Also, 
preparation  of  forms  requires  the  user  to  be  able  to  recognize 
and  identify  symbolic  information.  However,  tasks  in  this 


271 


category  are  not  relevant  to  the  present  purposes,  because 

they  do  not  require  a  device  for  training. 

Task  selection.  Both  classification  schemes  were  used  to 
select  six  45E10  tasks  for  further  study.  The  sample  of  tasks 
includes  two  tasks  from  each  of  the  three  main  functional  areas 
in  turret  maintenance:  hydraulic  system,  fire  control  system, 
and  electrical  system.  The  sample  also  includes  two  tasks  from 
each  of  the  first  three  skill  categories  described  in  the 
previous  section:  procedures,  detection/classification,  and 
troubleshooting.  An  additional  stipulation  was  to  select  only 
duty  position  tasks  to  ensure  adequate  task  documentation.  The 
selected  tasks  are  summarized  in  Table  32. 

In  comparison  with  the  other  tasks,  the  troubleshooting  tasks 
are  quite  extensive  in  scope.  Both  the  turret  electrical  and 
fire  control  functional  systems  are  divided  into  multiple 
subsystems,  and  each  subsystem  may  evidence  multiple  fault 
symptoms.  Therefore,  it  became  necessary  to  limit  both  tasks  to 
a  single  symptom  in  a  single  subsystem.  In  addition,  there  are 
two  types  of  troubleshooting  procedures  to  consider  for  most 
symptoras.  Primary  troubleshooting  procedures  are  accomplished 
with  the  STE-Ml  electronic  test  equipment,  which  automatically 
identifies  faults  and  cues  an  appropriate  course  of  action. 
Alternate  troubleshooting  procedures  employ  the  more  primitive 
breakout  box  to  identify  specific  problems.  One  primary  and  one 
alternative  procedure  was  chosen  for  each  of  the  two 


Table  32  Tasks  Selected  for  Further  Study 


Title 

Number 

Skill  Type 

Functional 

System 

Bleed  Air  from 
Elevation  Mechanism 

091-489-1142 

Perform 

Procedures 

Hydraulic 

Remove/Install 

Turret  Networks  Box 

091-489-1010 

Perform 

Procedures 

Electrical 

Check  and  Adjust 

Drift 

091-489-1194 

Detect/ 

Classify 

Hydraulic 

Service  Commander's 
Extension  (GPSE) 

091-489-1198 

Detect/ 

Classify 

Fire  Control 

Troubleshoot  Turret 
Electrical  System 

091-489-1056 

Troubleshoot 

Electrical 

Troubleshoot  Fire 
Control  System 

091-489-1256 

Troubleshoot 

Fire  Control 
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troubleshooting  tasks.  Selected  symptoms  and  procedures  are 
described  in  more  detail  below. 

1.  Troubleshoot  the  Turret  Electrical  System.  The  turret 
electrical  system  is  divided  into  five  subsystems:  vehicle/ 
turret  power  control,  fire  control,  fan  assembly,  turret 
breaker  monitor,  and  panel  lights  and  domelights.  The  STE-Ml 
applies  to  only  the  first  two  of  these  five  subsystems.  The 
vehicle/turret  power  control  (V/TPC)  subsystem  was  chosen  over 
the  firing  circuits  subsystem  to  conceptually  distinguish  this 
task  from  the  next  one  entitled  "Troubleshoot  Fire  Control 
System."  There  are  five  sets  of  symptoms  that  are  associated 
with  the  V/TPC  subsystem,  each  of  which  may  be  addressed  by 
either  a  primary  or  an  alternate  troubleshooting  procedure. 

To  illustrate  a  primary  troubleshooting  procedure,  the  first 
symptom  entitled  "Vehicle  Master  Power  Cannot  Be  Turned  On 
From  Commander's  Control  Panel"  (V/TPC-1)  was  chosen.  For  the 
alternate  troubleshooting  procedure,  the  second  symptom 
entitled  "Vehicle  Master  Power  Cannot  Be  Turned  Off  from 
Commander's  Control  Panel"  (V/TPC-2)  was  chosen. 

2.  Troubleshoot  the  Fire  Control  System.  The  fire  control  system 
is  divided  into  six  subsystems:  auto  self  test,  computer  and 
azimuth/elevation,  gunner's  primary  sight  defroster,  gunner's 
auxiliary  sight  reticle,  laser  range  finder,  and  thermal 
imaging  system.  The  auto  self  test  subsystem  (ASTS)  was 
chosen  because  it  was  the  only  one  to  which  both  primary  and 
alternate  troubleshooting  procedures  applied.  To  illustrate  a 
primary  troubleshooting  procedure,  the  symptom  entitled  "FIRE 
CONTROL  MALE  Light  and  F  Symbol  Come  On.  Computer  Manual  Self 
Test  Shows  No  Failure"  (ASTS-1)  was  chosen.  For  the  alternate 
troubleshooting  procedure,  the  symptom  entitled  "FIRE  CONTROL 
MALE  Light  Does  Not  Come  On  When  A  Harness  Is  Disconnected  Or 
When  PANEL  LIGHTS  TEST  Pushbutton  Is  Pressed"  (ASTS-2)  was 
chosen. 

Current  Training  Devices 

In  addition  to  classroom  instruction,  instructors  use  an 
actual  Ml  tank  and  two  different  computer-based  training  devices 
to  provide  training  on  turret  mechanic  tasks.  The  tank  is  used 
primarily  to  train  tasks  related  to  the  turret  hydraulic  system, 
whereas  the  computer-based  devices  train  troubleshooting  of  both 
the  fire  control  and  electrical  systems.  The  computer-based 
devices  are  described  in  more  detail  below.  This  section 
concludes  with  a  brief  summary  of  the  extent  to  which  the  devices 
simulate  each  of  the  selected  tasks. 

Turret  trainer.  The  Ml  Tank  Turret  Organizational  Maintenance 
Trainer  (Device  17-67)  or,  more  simply,  the  turret  trainer  is  a 
full-scale,  three-dimensional  representation  of  an  Ml  turret. 

The  major  physical  components  of  the  turret  trainer  are  (a)  the 
turret  stand,  which  is  a  welded  frame  structure  that  supports  the 
Ml  turret;  (b)  the  viewing  platform,  which  is  a  platform  mounted 
at  the  level  of  the  turret  ring  for  holding  up  to  13  viewers;  and 
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(c)  the  instructor's  control  station,  from  which  the  instructor 
can  control  the  simulation  (TM  9-6910-242-10) .  Each  of  the 
turret  trainer's  75  exercises  simulates  a  single  set  of  fault 
symptoms  for  the  student  to  troubleshoot.  The  exercises  call  for 
the  student  to  identify  the  symptoms  and  to  use  the  technical 
manuals  to  correct  the  fault.  Many  of  the  procedural  tasks 
(e.g.,  remove  and  replace  components)  can  be  performed  on  this 
trainer  exactly  as  they  would  be  on  the  actual  tank. 

Panel  trainer.  The  Ml  Turret  Organizational  Maintenance 
Troubleshooting  Trainer  (Device  17-68/1)  or,  more  simply,  the 
panel  trainer  provides  a  two-dimensional  representation  of  turret 
components.  The  major  components  include  (a)  the  display  panel, 
which  is  an  upright  plane  surface  containing  a  combination  of 
pictures,  controls,  and  displays  that  simulate  Ml  tank  system 
components,  controls,  and  displays;  (b)  the  viewer  assembly, 
which  is  a  random-access  slide  projector  that  presents  pictures, 
word  messages,  or  a  combination  of  both  in  accordance  with 
actions  taken  on  the  display  panel  and  control  console;  and  (c) 
and  instructor/ operator  station,  which  includes  a  CRT  terminal 
and  control  console  for  both  input  and  output  (TM  9-6910-243- 
10).  Like  the  turret  trainer,  each  of  the  panel  trainer's  63 
exercises  simulates  a  single  set  of  symptoms  for  the  student  to 
troubleshoot.  Unlike  the  turret  trainer,  the  panel  trainer's 
components  are  represented  by  two-dimensional  pictures  whose 
functions  are  simulated  by  rotary  and  pushbutton  switches. 

Whereas  the  representations  of  the  STE-Ml  and  the  breakout  box 
are  not  fully  functional  measuring  devices,  they  do  provide 
appropriate  readings  within  an  exercise. 

Simulation  of  selected  tasks.  Some  of  the  tasks  selected  for 
further  study  may  be  trained  on  the  two  computer-based  devices, 
while  others  are  either  partially  supported  by  the  devices  or  are 
not  supported  at  all.  Table  33  provides  a  qualitative  summary 
and  comparison  of  the  device's  capabilities  in  this  regard. 

Resident  Data  Requirements 

Resident  data  refer  to  those  data  that  are  relatively 
independent  of  the  specific  OSBATS  application.  This  statement 
does  not  mean  that  resident  data  variables  will  be  the  same  over 
all  training  domains.  We  would  not  expect  to  find  the  same 
concerns  over  the  environmental  visual  display  in  Armor  turret 
maintenance  that  was  so  critical  in  Aviation  flight  training. 
Nevertheless,  we  would  expect  most  of  the  resident  data  variables 
that  are  relevant  for  training  the  pilot  of  the  AH-1  helicopter 
to  be  relevant  for  other  helicopters,  as  well.  Differences  among 
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Table  33 


Summary  ot  Capability  of  Devices  to  Support  Training 
on  Selected  Tasks 


Task 

Procedure :  Symptom 

Turret  Trainer 

Panel  Trainer 

Bleed  Air  from 
Elevation  Mechanism 

Cannot  be 
perfoirmed. 

Cannot  be 
performed. 

Remove/Install  Turret 
Networks  Box  (TNB) 

Performed  as 
actual  tank. 

in 

Simulated  by 
button  pushes. 

Check  and  Adjust 

Drift 

Cannot  be 
performed. 

Cannot  be 
performed. 

Service  Commander's 
Extension  (GPSE) 

Performed  as 
in  actual  tank. 

Cannot  be 
performed . 

Troubleshoot  Turret  Electrical  System 

Primary:  V/TPC-1 

Simulated  by  Exercise 
Numbers  33  and  56. 

Simulated  by 
Exercise  Number  1. 

Alternate;  V/TPC-2 

Cannot  be 
performed. 

Cannot  be 
performed. 

Troubleshoot  Fire  Control  System 

Primary;  ASTS-1 

Simulated  by 
Number  8 . 

Exercise 

Simulated  by 
Exercise  Number  41 

Primary:  ASTS-2 

Simulated  by 
Number  69. 

Exercise 

Cannot  be 
performed. 

helicopters  would  reflect  differences  in  the  missions  performed 
by  the  aircraft,  or  differences  in  the  special  equipment  on  the 
aircraft.  Similarly,  the  resident  data  for  the  Turret  Mechanic's 
course  for  the  MlAl  Tank  should  easily  generalize  to  other  tanks, 
and,  with  some  changes,  to  other  ground  vehicles. 

The  three  types  of  data  are  addressed  in  the  resident  data 
requirements  include  training  device  data,  fidelity  dimension 
data,  and  instructional  feature  data. 

Fidelity  Dimension  Data 

Fidelity  dimensions  and  levels.  In  thinking  about  training 
and  fidelity,  the  system  that  the  students  will  use  should  be 
clearly  distinguished  from  the  environment  in  which  that  system 
will  be  used.  For  the  flight  trainer,  the  system  is  clearly  the 
helicopter,  and  the  environment  is  the  terrain  and  enemy.  Five 
vicuc-l  variables  and  three  variables  related  to  terrain  size  and 
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visual  special  effects  were  used  for  the  environment.  Three 
variables  related  to  motion  and  audio  simulation  addressed  the 
system. 


In  a  maintenance  trainer  for  the  M-1  tank,  the  system  is  not 
the  tank,  but  rather  the  standard  test  equipment  and  breakout 
box.  Standard  test  equipment  and  a  breakout  box  are  so  basic  and 
inexpensive  that  fidelity  variations  of  these  items  within  the 
OSBATS  model  are  not  likely  to  be  productive.  As  shown  in  the 
last  variable  of  Figure  10,  this  test  equipment  will  either  be 
available  or  not. 

The  M-1  tank  is  the  students'  training  environment  in  the 
turret  mechanic  course.  The  key  elements  of  this  environment 
from  a  training  standpoint  are  (a)  the  visual  cues  that  are 
necessary  for  finding  components  and  determining  the  best  route 
of  access  for  testing  and  replacement  and  (b)  the  actual  physical 
representations  required  for  using  the  test  equipment  while 
performing  the  task.  Four  vaiiables  address  these  elements  using 
the  major  systems  of  the  turret:  (a)  motion/hydraulics  system, 

(b)  electrical  system,  (c)  fire  control  system,  and  (d)  main 
weapon  system.  The  hydraulics  system  has  related  motion 
requirements  for  the  turret  and  gun  tube  so  that  the  proper 
visual  cues  can  be  presented.  In  every  case  (except  the  main 
weapon  system)  the  fidelity  levels  progress  from  none  through 
very  high  quality  replicas  of  components  within  each  of  the 
systems.  Intermediate  fidelity  levels  include  two-dimensional 
panel  trainer  representations  with  pictures  and  controls  (e.g., 
switches) .  Such  a  fidelity  level  for  a  given  system  would  be 
useful  for  training  the  sequence  of  procedures  but  would  provide 
no  visual  cues  or  practice  with  the  test  equipment. 
Troubleshooting  tasks  would  not  be  trainable  with  this  level  of 
fidelity.  An  enhanced  intermediate  level  of  fidelity  is  termed 
interactive  graphics;  the  presentation  of  high  quality,  two- 
dimensional  visual  cues  as  controlled  by  the  student  using  a 
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Figure  10.  Fidelity  dimensions  and  levels  for  armor  turret 
maintenance  training. 
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computer-driven,  video-disc  based  system,  for  example.  Again  no 
practice  with  the  test  equipment  is  provided. 


Advanced  levels  of  fidelity  for  all  four  system  related 
variables  include  actual  physical  representations  for  portions  of 
each  system.  These  representations  would  be  included  in  the 
model  in  the  order  of  their  contributions  to  the  training  of  the 
tasks  identified  by  the  turret  mechanic  Program  of  Instruction. 
The  hydraulic  system  includes  the  following  levels  beyond  the 
two-dimensional  representations:  limited  movement  of  the  turret 
and  gun  tube  via  an  electronic  system  so  that  the  proper 
procedures  and  visual  cues  are  possible  from  mock-ups  of  the 
hydraulic  system;  the  actual  hydraulic  subsystems  required  for 
traversing,  troubleshooting  and  maintaining  the  turret  and  then 
for  elevating  the  gun  tube;  and  finally  realistic  representations 
of  the  hydraulic  system  controls  so  that  the  students  learn 
where  and  how  the  hydraulic  system  is  activated. 

Enhancements  for  the  electrical  system  permit  removal  and 
replacement,  troubleshooting  and  testing  of  the  turret  networks 
box,  main  gun  safety  switch,  gas  particulate  filter  system,  and 
the  ready  ammunition  door.  Physical  representations  of  the  fire 
control  system  include  the  laser  range  finder,  commander's  sight, 
computer  electronics  unit,  hydraulic  turret  valves,  and 
accumulators.  Elements  of  the  main  weapon  system  that  would  be 
evaluated  include  the  breech  operating  mechanism  and  the  firing 
circuit  of  the  commander's  weapon  system. 

The  ability  to  simulate  various  faults  within  the  M-1  turret 
system  is  analogous  to  the  capability  within  a  aviation  training 
device  to  overlay  various  visual  images  to  simulate  points  (e.g. , 
cultural  lights  and  moving  vehicles)  and  areas  (e.g.,  smoke  and 
rotor  wash) .  Levels  of  fidelity  include  individually  simulating 
a  set  of  faults  such  as  a  short  in  the  turret  networks  box, 
followed  by  simulating  several  of  the  faults  simultaneously. 

Finally  there  are  several  auditory  cues  that  can  be 
represented  to  aid  in  troubleshooting  and  maintenance  tasks: 
hydraulic  system  noises,  solenoid  clicks,  and  the  rubbing  and 
scraping  of  moving  parts.  Table  34  lists  all  seven  of  the 
fidelity  dimensions  and  their  levels  that  have  been  discussed. 

Technical  performance  and  cost  data.  Data  concerning 
technical  performance  and  development  costs  for  these  dimensions 
and  levels  of  fidelity  would  be  developed  exactly  the  same  as 
would  be  done  for  the  aviation  example,  subject  matter  experts 
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Table  34  Dimensions  and  Levels  of  Fidelity  for  a  Turret 

Maintenance  Trainer 

Dimension  1  -  Motion/Hydraulic  System 
Level  1  -  None 

Level  2  -  Pictures  and  Controls  on  a  Panel 
Level  3  -  Automated,  2-dimensional,  interactive  graphics 
Level  4  -  Limited  movement  via  electronics  and  simulated 
hydraulics 

Level  5  -  Actual  hydraulics  for  traversing  the  turret 
Level  6  -  Plus  actual  hydraulics  for  elevating  the  gun  tube 
Level  7  -  Plus  actual  hydraulic  system  controls 

Dimension  2  -  Electrical  System 
Level  1  -  None 

Level  2  -  Pictures  and  Controls  on  a  Panel 

Level  3  -  Automated,  2 -dimensional ,  interactive  graphics 

Level  4  -  Actual  turret  networks  box 

Level  5  -  Plus  actual  main  gun  safety  switch 

Level  6  -  Plus  actual  gas  particulate  filter  system 

Level  7  -  Plus  actual  ready  ammunition  door 

Dimension  3  -  Fire  Control  System 
Level  1  -  None 

Level  2  -  Pictures  and  Controls  on  a  Panel 

Level  3  -  Automated,  2-dimensional,  interactive  graphics 

Level  4  -  Actual  laser  range  finder 

Level  5  -  Plus  actual  commander's  sight 

Level  6  -  Plus  actual  computer  electronics  system 

Level  7  -  Plus  actual  hydraulic  turret  valves 

Level  8  -  Plus  actual  accumulators 

Dimension  4  -  Main  Weapon  System 
Level  1  -  None 

Level  2  -  Actual  breech  operating  mechanism 
Level  3  -  Plus  actual  CWS  firing  circuit 

Dimension  5  -  Simulated  Faults 
Level  1  -  None 
Level  2  -  Single  Faults 
Level  3  -  Multiple  Faults 

Dimension  6  -  Auditory  Cues 
Level  1  -  None 
Level  2  -  Hydraulic  sounds 
Levtil  2  -  Plus  solenoid  clicks 
Level  4  -  Plus  moving  metal  parts 

Dimension  7  -  Test  Equipment 
Level  1  -  None 

Level  2  -  Actual  STE-Ml  and  Break-out  Box 
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(SMEs)  would  specify  the  technical  performance  indices  for  each 
level  of  each  dimension.  Then  cost  data  from  previous  training 
devices  or  estimates  from  engineers  would  be  used  to  develop 
statistical  estimates  for  the  parameters  of  the  cost  functions. 

Fidelity  rule  base.  The  development  of  a  fidelity  rule  base 
would  require  SME  research,  just  as  it  did  for  the  aviation 
example.  We  expect  that  it  should  be  relatively  easy  to  develop 
some  of  the  rules  in  the  rule  base.  For  example,  if  a  task 
requires  removing  a  component,  then  we  would  expect  that  the  task 
could  not  be  trained  on  a  training  device  unless  the  device 
represented  that  component  so  that  it  could  be  removed  and 
replaced.  However,  rules  for  troubleshooting  tasks  are  likely  to 
be  more  complex,  because  of  the  extensive  branching  involved  in 
these  tasks. 

Instructional  Feature  Data 


Instructional  features.  In  contrast  to  aviation  tasks, 
maintenance  tasks  do  not  require  a  high  degree  of  continuous 
control,  nor  do  they  possess  significant  time-sharing  demands. 
Consequently,  some  of  the  instructional  features  appropriate  for 
aviation  would  be  excluded  from  armor  maintenance,  including  (a) 
parameter  freeze,  (b)  record/ replay,  and  (c)  remote  replay. 

Both  trainers  include  a  system  freeze  feature.  This  fact  is 
consistent  with  the  current  instructional  features  rule  base  that 
states  that  system  freeze  is  appropriate  for  long  procedures. 

The  turret  trainer  also  includes  a  performance  indicators 
feature.  As  presently  implemented,  this  feature  allows  the 
instructor  to  encode  the  turret  component  that  a  student 
replaces.  This  feature  is  required  because  the  system  computer 
cannot  detect  the  student's  responses.  In  contrast,  the  panel 
trainer  does  not  include  this  feature,  presumably  because  the 
student's  limited  repertoire  of  permissible  responses  (button 
pushes,  knob  turns)  are  all  computer  detectable.  This  result  is 
also  consistent  with  the  instructional  feature  rule  base. 

Of  the  21  instructional  features  currently  considered  by  the 
OSBATS  model,  the  following  instructional  features  are  present  in 
some  form  in  one  or  both  of  the  turret  maintenance  trainers. 

1.  Automated  Performance  Measurement 

2.  Performance  Indicators 

3 .  Procedure  Monitoring 

4 .  Total  System  Freeze 

5.  Initial  Conditions 

6.  Scenario  Control 

7.  Simulation  Demonstration  (manual) 

Although  these  instructional  features  serve  the  same  function  as 
comparable  ones  in  aviaLion  tiaiuing  devices,  uncir 
implementation  may  be  significantly  different  in  the  current 
context  from  the  implementation  in  aviation.  Thus,  one  should 
take  care  when  extrapolating  data  variables  across  domains. 
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The  following  instructional  features  from  aviation  training 
appear  to  be  inappropriate  in  the  armor  maintenance  context. 

1.  Simulator  Record/Playback 
2  .  Remote  Graphics  Replay 

3 .  Crash  Override 

4 .  Parameter  Freeze 

5.  Flight  System  Freeze 

6.  Position  Freeze 

7.  Real-Time  Simulation  Variables  Control 

The  remaining  instructional  features  appear  to  be  appropriate  for 
some  tasks,  although  they  are  not  implemented  in  either  existing 
trainer. 

1.  Automated  Performance  Alerts 

2 .  Augmented  Feedback 

3 .  Augmented  Cues 

4.  Reset/Reposition 

5.  Adjunct  Computer-Assisted  Instruction 

6.  Automated  Adaptive  Training 

7.  Automated  Cueing  and  Coaching 

We  did  not  uncover  additional  instructional  features  that  were 
not  considered  in  aviation. 

Instructional  feature  cost.  We  anticipate  that  instructional 
feature  cost  may  be  estimated  using  the  same  procedures  as  would 
be  appropriate  for  aviation  training.  In  this  respect,  we  would 
also  expect  many  of  the  same  difficulties  in  determining 
instructional  feature  cost  for  armor  training  devices  that  have 
been  experienced  in  assessing  costs  for  aviation  training 
devices.  We  do  not  think  that  it  would  be  feasible  to  determine 
costs  by  extrapolating  from  aviation. 

Instructional  feature  benefit  weight.  The  instructional 
feature  benefit  weights  were  based  on  a  survey  of  instructors 
regarding  the  extent  to  which  they  used  various  instructional 
features.  We  anticipate  that  this  procedure  would  work  in  the 
armor  maintenance  domain,  as  well.  However,  since  there  are 
fewer  training  devices  available,  it  would  be  more  difficult  to 
obtain  as  large  a  sample  of  respondents  on  which  to  base  the 
feature  weights.  In  addition,  some  features  may  not  be 
implemented  in  any  existing  training  device.  For  these  features, 
the  feature  weight  would  be  assessed  by  comparison  to  existing 
features,  or  by  asking  instructors  if  they  would  use  such  a 
feature  if  it  existed. 

Instructional  feature  rules.  Many  of  the  instructional 
feature  rules  appear  to  be  the  same  in  the  armor  domain  as  they 
are  for  aviation.  We  anticipate  that  a  closer  examination  of  how 
instructional  features  are  used  in  armor  maintenance  training 
would  reveal  uses  that  were  not  considered  in  our  analysis  of 
aviation  training.  These  new  considerations  could  be  represented 
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as  additional  instructional  feature  rules.  We  think  it  likely 
that  most  of  the  new  rules  would  apply  to  aviation  as  well  as 
armor. 


Training  Device  Data 

Training  device  data  are  descriptions  of  existing  or 
hypothetical  training  devices  according  to  cost,  and  according  to 
the  fidelity  dimensions  and  instructional  features  defined  above. 
The  procedures  for  defining  and  collecting  these  data  should  be 
the  same  as  those  used  in  aviation  training. 


Task  Data  Requirements 

Task  data  requirements  include  the  data  that  are  unique  to 
each  application  of  the  OSBATS  model.  These  data  include  task 
learning  points,  task  simulation  requirements,  task  training 
hours  and  costs,  and  data  for  the  instructional  feature  and 
fidelity  rule  bases.  For  the  most  part,  there  does  not  appear  to 
be  any  special  problems  in  obtaining  these  data  in  the  armor 
domain.  Some  comments  on  specific  data  categories  follow. 

1.  Task  learning  points.  The  procedure  used  to  scale  entry  level 
performance  and  the  performance  standard  will  probably  be 
somewhat  different  for  aviation  maintenance  tasks  than  for 
aviation  operations  tasks,  because  of  the  differences  in  the 
kinds  of  skills  required.  We  anticipate  that  when  scaling 
methods  are  defined,  there  will  be  several  different  kinds  for 
continuous  control,  procedures,  cognitive  tasks,  and  so  forth. 
Thus,  the  scaling  procedure  will  probably  depend  more  on  skill 
type  than  training  domain. 

2.  Simulation  requirements.  The  reasons  for  simulated  training 
are  probably  somewhat  different  in  the  current  domain  than 
they  are  in  aviation.  For  example,  damage  to  equipment  is 
probably  a  somewhat  greater  concern  here,  while  existence  of 
training  ranges  is  of  no  concern  at  all.  This  result  would 
suggest  the  need  to  develop  a  comprehensive  list  of  reasons 
for  simulation,  and  a  method  of  aggregating  ratings  of  these 
reasons  that  is  independent  of  domain. 

3.  Data  for  fidelity  rule  base.  Since  the  rule  base  has  not  been 
developed,  its  data  requirements  are  unknown.  It  seems  likely 
that  these  requirements  will  be  somewhat  simpler  than  the 
requirements  in  aviation,  because  it  is  not  necessary  to 
specify  the  requirements  for  a  visual  system.  Specification 
of  the  visual  system  accounts  for  most  of  the  rules  in  the 
existing  fidelity  rule  base. 

4.  Data  for  instructional  feature  rule  base.  It  seems  likely 
that  the  instructional  feature  rule  base  will  be  the  same 
across  domains.  Hence,  the  data  requirements  will  also  be  the 
same. 
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5.  Training  hours  and  costs.  These  data  should  be  no  different 
in  this  domain  than  they  were  in  aviation. 


Comparison  of  the  Model  Process 

Our  analysis  did  not  uncover  any  changes  in  the  general  OSBATS 
model  process  that  would  be  required  to  make  it  applicable  for 
optimization  of  armor  maintenance  training  systems.  However, 
there  are  several  differences  in  the  two  domains  we  have  analyzed 
that  have  implications  on  the  operation  of  the  model. 

The  first  difference  is  in  the  complexity.  The  aviation 
training  example  involved  more  tasks,  more  fidelity  dimensions, 
and  greater  variety  in  the  skills  trained.  This  apparent 
difference  is  partly  illusory,  however.  The  turret  mechanic  must 
be  able  to  perform  a  far  greater  number  of  tasks  than  are  covered 
in  the  Program  of  Instruction.  Many  of  these  are  listed  as 
"related  tasks,"  and  are  not  trained  under  the  assumption  that 
the  ability  to  perform  these  tasks  will  transfer  from  other 
tasks.  If  the  related  tasks  were  included  in  the  analysis,  then 
there  would  be  potential  implication  on  the  OSBATS  model,  because 
of  the  possibility  of  transfer  between  tasks,  which  is  not 
currently  considered  in  the  model. 

A  second  difference  is  in  the  methods  that  must  be  used  to 
determine  fidelity  requirements.  The  analysis  of  visual  fidelity 
requirements  in  the  fidelity  rule  base  for  aviation  involves  a 
detailed  analysis  of  the  kinds  of  activities  required  to  perform 
a  task,  where  there  activities  include  such  specific  actions  such 
as  estimating  altitude  or  range,  or  detecting  distant  targets. 

It  seems  that  the  analysis  for  the  turret  mechanic  would  not  be 
as  complex,  and  would  not  require  the  same  depth  of  knowledge 
about  the  task.  Thus,  it  might  be  more  likely  that  the  engineer 
who  is  the  user  of  the  OSBATS  model  might  be  able  to  provide  the 
data  for  the  fidelity  rule  base  in  maintenance  problem,  while 
considerably  greater  subject-matter  knowledge  would  be  required 
to  provide  comparable  data  for  aviation  maintenance. 

A  third  difference  is  that  the  reasons  for  simulation  are 
somewhat  different  in  the  two  domains.  This  difference  will  have 
an  impact  on  the  kinds  of  factors  that  are  considered  in 
evaluating  the  benefits  that  may  be  derived  from  device-based 
training. 


Summary  and  Conclusions 

The  overall  conclusion  of  the  analysis  is  that  the  OSBATS 
model  should  be  directly  applicable  to  the  M-1  Abrams  Turret 
Mechanic  Course.  No  changes  would  be  required  in  the  general 
model  processes  and  organization  of  modules.  One  specific  change 
within  the  Simulation  Configuration  Module  would  likely  be 
required.  However,  application  of  OSBATS  to  the  new  domain  would 
require  considerable  development  of  resident  data,  particularly 
fidelity  dimension  data. 


282 


The  difficulty  of  the  modeling  process  revolves  around  the 
specification  of  the  fidelity  dimensions  and  levels.  The 
complexity  of  the  model  is  a  function  of  both  the  system  being 
used  by  the  students  and  the  environment  in  which  the  students 
use  that  system.  For  the  flight  trainer  the  syste.m  was  the 
helicopter,  which  was  considerably  more  complex  than  the  test 
equipment  of  the  maintenance  trainer.  However,  the  environment 
of  the  maintenance  trainer  was  a  complex  tank  turret,  in  some 
ways  as  complex  than  the  environment  of  the  aviation  training 
device.  The  process  of  breaking  the  system  and  environment  into 
dimensions  and  then  defining  levels  of  fidelity  for  each  remains 
an  art,  but  the  process  is  much  better  understood  now  that  it  has 
been  completed  twice  and,  we  believe,  codifiable  in  the  near 
future.  What  is  required  is  feedback  from  maintenance  trainers 
on  the  dimensions  and  levels  described  above  and  the  development 
of  dimensions  and  levels  for  at  least  one  more  application. 

Once  the  fidelity  dimensions  and  levels  are  specified,  the 
model  process  proceeds  systematically.  The  OSBATS  data  base  must 
be  developed  around  the  definitions  of  these  dimensions  and 
levels.  All  of  the  data  elements  are  defined;  most  of  the  data 
must  be  developed  with  the  support  of  SMEs. 


SUMMARY  AND  CONCLUSIONS 


The  high  cost  of  training  using  actual  weapon  systems  and  the 
expanded  capability  of  training  technology  have  increased  the 
potential  value  of  simulation-based  training.  However,  the 
complexity  of  weapon  systems  and  their  associated  training 
systems  has  made  the  process  of  designing  training  devices  much 
more  difficult.  The  process  of  formulating  a  cost-effective 
training-de-^'ice  concept  requires  many  tradeoff  analyses  that 
compare  the  cost  and  effectiveness  of  alternative  design 
concepts. 

The  OSBATS  model  aids  the  system  engineer  responsible  for 
formulating  a  training-device  concept.  The  goal  of  the  OSBATS 
model  is  to  provide  methods  to  produce  training  device  designs 
that  meet  the  training  requirements  at  the  minimum  cost.  The 
model  helps  the  system  engineer  to  design  training  devices  and  to 
incorporate  them,  along  with  actual  equipment,  in  a  training 
system. 

This  report  has  described  the  OSBATS  model  in  detail  from  the 
viewpoint  of  both  the  model  developer  and  the  user.  The  model 
description  has  described  the  input  data  requirements,  the 
specific  model  processes,  and  the  products  of  the  model  analyses. 
In  addition,  the  report  has  discussed  the  results  of  the 
formative  evaluation  activities  conducted  in  this  effort. 

In  this  section  of  the  report,  vie  will  summarize  the 
capabilities  of  the  OSBATS  model,  highlighting  both  the 
accomplishments  made  by  this  effort  and  the  gaps  in  our  knowledge 
that  need  to  be  addressed  by  future  research.  Our  summary  will 
focus  on  three  issues.  First,  we  will  describe  the  analyses 
performed  by  the  OSBATS  model,  paying  particular  attention  to  how 
these  analyses  may  be  used  to  meet  specific  needs  of  the  system 
engineer.  Second,  we  will  highlight  what  we  think  are  the  major 
accomplishments  of  the  development  effort.  Finally,  we  will 
summarize  the  activities  that  are  required  for  validation  and 
technology  transfer. 

Summary  of  OSBATS  Functions 

The  OSBATS  system  provides  tools  to  aid  the  tradeoff  analyses 
required  to  design  cost-effective  training  devices.  The  model 
allows  design  engineers  to  consider  training  effectiveness 
seriously  when  they  develop  a  training-device  design  concept.  It 
provides  an  interactive  environment  that  allows  the  engineer  to 
consider  many  more  alternative  designs  than  would  be  possible 
without  the  model.  Using  the  OSBATS  model,  the  user  can  perform 
comparative  analyses  that  identify  cost  drivers,  produce  and 
evaluate  alternative  training-device  design  concepts,  and  specify 
cost-efficient  ways  to  use  training  devices  to  meet  the  training 
requirements. 

Five  modules  interact  to  help  the  engineer  develop  and 
evaluate  training-device  concepts.  The  engineer  can  use  the 
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modules  singly  or  in  combination  to  address  a  wide  variety  of 
training-device  design  issues.  The  following  list  describes  some 
of  the  analyses  that  can  be  performed  using  the  OSBATS  model. 

1.  Screen  training  requirements  to  determine  which  requirements 
can  be  met  most  appropriately  using  some  kind  of  training 
device. 

2.  Identify  tasks  that  can  be  trained  adequately  using  a  simple, 
inexpensive  training  device. 

3.  Compare  the  thousands  of  potential  training-device  design 
options  to  determine  which  ones  meet  the  specific  task 
training  requirements  at  the  lowest  cost. 

4.  Examine  the  minimum  fidelity  levels  required  to  train  a  task, 
based  on  the  specific  activities  performed  as  a  part  of  the 
task. 

5.  Determine  which  instructional  support  features  are  needed  to 
maximize  the  efficiency  with  which  the  training  requirements 
may  be  met  on  a  specific  training  device. 

6.  Compare  the  cost  effectiveness  of  training  conducted  using  a 
sophisticated,  lull-mission  simulator  with  training  conducted 
using  a  combination  of  simpler,  part-mission  simulators. 

7.  Compare  the  cost-effectiveness  of  a  design  proposed  by  the 
user  or  other  individual  with  a  design  of  the  same  cost 
recommended  by  the  model . 

8.  Determine  how  training  time  should  be  allocated  among  training 
devices  and  actual  equipment. 

9.  Investigate  the  effect  of  limited  availability  of  actual 
equipment  or  a  training  device  on  the  training  time  and  cost 
required  to  meet  the  training  requirements. 

Sijnificant  Accomplishments  of  This  Research 

Our  three-year  effort  to  develop  the  OSBATS  model  has  produced 
several  advancements  in  the  state  of  the  art  for  training-device 
optimization.  These  advancements  build  on  the  results  of 
previous  research  and  existing  models.  The  following  paragraphs 
summarize  the  most  significant  accomplishments,  which  distinguish 
the  OSBATS  model  from  predecessor  models. 

First,  The  OSBATS  system  provides  a  consistent  approach  to 
address  a  variety  of  training-device  design  problems.  Its 
consistency  comes  from  its  top-down  design,  and  its  coordinated 
use  of  cost-benefit  optimization  in  each  of  its  components.  Each 
module  addresses  one  aspect  of  the  training-system  design  process 
and  recommends  an  optimal  choice  by  considering  the  factors  that 
affect  the  costs  and  benef-its  relevant  to  the  process.  The 
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modules  share  common  concepts,  such  as  learning  rates,  task 
weigiits,  and  cost  elements,  to  ensure  consistent  results. 

The  design  of  the  OSBATS  system  provides  the  flexibility 
required  to  accommodate  the  complex  interactions  involved  in 
training-system  design.  This  approach  captures  tne  inherently 
iterative  nature  of  the  training-system  design  process  as 
described  early  in  this  report.  The  model  provides  methods  that 
allow  the  results  of  analyses  using  any  OSBATS  modulti  to  provide 
information  used  by  other  modules.  Thus,  the  model's  modular 
structure  allows  easy  analysis  and  refinement  of  results. 

The  characteristic  of  the  OSBATS  model  that  most  distinguishes 
it  from  its  predecessors  is  its  emphasis  on  training-device 
design.  The  training-device  design  modules  of  the  OSBATS  model 
allow  the  user  to  in  estigate  and  compare  many  design  options. 

The  design  engineer  may  use  the  results  of  these  modules  to 
determine  which  fidelity  and  instructional  feature  alternatives 
should  be  included  in  the  training-device  design,  based  on  the 
training  requiremercs.  All  other  existing  training-development 
models  emphasize  /aluati.on.  Those  models  allow  the  user  to 
evaluate  a  single  training-device  design.  Application  of  other 
models  to  a  large  number  of  alternative  designs  would  be  overly 
burdensome  on  the  design  engineer.  Thus,  the  0:13ATS  model  has 
opened  up  a  stage  of  the  training-device  design  process  to  the 
benefits  of  analytic  modeling. 

The  OSBATS  model,  unlike  most  others,  aggregates  cost  and 
effectiveness  estimates  to  develop  recommendations  based  on  a 
effectiveness/cost  ratio.  Other  models  apply  a  benefit  analysis 
followed  by  a  cost  analysis.  For  example,  Kribs,  Simpson,  and 
Mark  (1S83) ,  in  their  review  of  medium  selection  models, 
identified  five  subtasks  that  were  common  to  these  models.  These 
subtasks  included  a  ranking  of  training  media  for  training 
ef fectivene;^s  followed  by  a  cost  tradeoff  analysis  used  to 
perform  the  final  selection.  In  a  similar  fashion,  the 
instructional  support  feature  guidelines  developed  for  the  Air 
Force  (Logicon,  1985)  specify  a  benefit  analysis  followed  by  an 
analysis  of  technology  and  cost  considerations.  The  integrated 
effectiveness/cost  analysis  provided  by  OSBATS  is  superior  to 
methods  that  perform  sequential  effectiveness  and  cost  analyses, 
in  that  the  latter  methods  tend  to  reject  options  that  offer 
moderate  benefit  at  a  low  cost  in  favor  of  options  that  offer 
high  benefit  at  a  high  cost.  Usually  more  overall  effectiveness 
can  be  obtained  within  a  cost  budget  with  several  moderately 
effective,  but  inexpensive  opticns,  than  with  one  or  two  highly 
effective,  but  expensive  options. 

In  developing  the  OSBATS  model,  we  have  produced  the  following 
other  advancements  in  specific  areas  of  training-system  modeling. 


1.  We  have  extended  the  framework  for  training-device 

optimization  initially  proposed  by  Roscoe  (1971)  to  consider 
the  impact  of  constraints  on  the  use  of  training  device  or 
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actual  equipment  on  the  time  or  cost  required  to  meet  training 
requirements. 

2.  We  have  developed  new  procedures  to  determine  task  fidelity 
and  instructional  feature  requirements  from  descriptions  of 
the  activities  involved  in  the  tasks. 

3 .  We  have  developed  new  methods  to  cluster  tasks  according  to 
their  needs  for  simulation  and  their  requirements  for  a 
sophisticated  simulation  capability. 

Needs  for  Future  Model  Development 

The  OSBATS  model  has  been  completely  specified  and  prototype 
software  has  been  developed.  However,  further  work  is  needed 
before  the  model  is  transferred  to  the  user  community.  The 
required  activities  include  model  expansion,  data  base 
development,  model  calibration  and  validation,  and  software 
enhancement.  Some  specific  needs  are  outlined  below. 

Technology  Transfer 

The  ultimate  goal  of  the  OSBATS  research  and  development 
effort  is  the  transfer  of  the  software  to  the  engineers 
responsible  for  training-device  concept  formulation.  However, 
the  current  version  of  the  system  is  not  sufficiently  developed 
to  allow  direct  transfer  to  users.  Barriers  to  technology 
transfer  come  from  both  limits  in  the  state  of  model  development, 
and  from  the  process  by  which  the  model  was  developed. 

The  current  OSBATS  data  base  supports  the  use  of  the  model 
over  a  limited  domain.  Although  the  specific  domain  of 
application  is  the  AH-1  training  course,  we  think  that  the  model 
should  be  applicable  with  only  minor  changes  to  most  training 
domains  involving  rotary-wing  aviation  operations.  Use  of  the 
model  outside  of  this  domain  requires  the  user  either  to  collect 
additional  data  from  subject-matter  experts  or  to  make 
assumptions  about  the  values  of  such  data  and  suffer  a  consequent 
loss  in  the  accuracy  of  the  model's  predictions.  Furthermore, 
operation  of  the  model  outside  of  the  domain  for  which  it  was 
originally  developed  will  probably  require  assistance  from  the 
model  developer,  the  programmer,  or  both,  to  tailor  the  model  to 
the  new  situation.  Although  our  evaluation  has  indicated  that 
the  model  processes  are  general,  we  expect  that  situations  will 
arise  in  future  applications  that  require  modifications  to  the 
model  or  software. 

Nevertheless,  it  should  be  stressed  that  it  is  possible,  with 
appropriate  assistance,  to  apply  the  model  to  a  wide  variety  of 
problems.  We  think  that  the  application  of  the  model  on  an 
actual  training  design  problem  should  be  a  high  priority.  The 
model  application  will  establish  a  working  relationship  between 
model  developers  and  model  users.  The  feedback  obtained  from 
model  users  will  provide  a  wealth  of  information  that  can  be  used 
to  improve  the  model.  In  addition,  we  are  confident  that  the 
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model  will  provide  the  engineer  insights  that  can  be  used  to 
produce  a  better  training  device  concept. 

The  initial  phase  of  the  OSBATS  development  process  was 
conducted  with  limited  interactions  with  the  eventual  model 
users.  The  engineers  were  used  primarily  to  evaluate  the 
software  and  to  provide  information  on  the  procedures  currently 
used  for  training-device  concept  formulation.  Future  development 
should  have  much  greater  involvement  by  the  engineers  who  will 
use  the  OSBATS  model  in  concept  formulation.  We  recommend  that 
future  development  efforts  include  a  mechanism  that  will  provide 
an  ongoing  dialogue  between  the  model  developers  and  potential 
users  to  tailor  the  model  to  user  needs,  increase  user  ownership 
of  the  model,  and  support  technology  transfer. 

Other  needs  for  future  model  development  support  the  need  for 
technology  transfer.  That  is,  new  model  capabilities,  more 
comprehensive  data  bases,  easy  data  collection  procedures,  and 
model  calibration  and  validation  will  all  increase  the  likelihood 
of  successful  technology  transfer,  as  well  as  offer  other 
enhancements  to  the  quality  of  the  OSBATS  model. 

Additional  Modeling  Capabilities 

We  envision  that  as  the  OSBATS  model  is  transferred  to  the 
training-device  design  engineers,  many  of  the  requirements  for 
additional  modeling  capabilities  will  come  from  the  user.  At 
this  stage  in  the  development  process,  we  have  received  some 
suggestions  from  potential  users;  other  ideas  have  come  from  our 
own  use  of  the  model.  The  following  list  briefly  describes 
several  possible  enhancements  to  the  OSBATS  model's  capability. 

1.  Development  of  new  task  clustering  methods  that  reflect  other 
rationale  for  partitioning  tasks,  such  as  similarity  of 
fidelity  requirements,  mission  phase,  and  so  forth.  One  of 
the  critical  early  decisions  in  training-device  design 
specifies  the  tasks  used  as  the  basis  of  the  training-device 
design.  The  current  Simulation  Configuration  Module  contains 
one  rationale  for  clustering  tasks.  Because  of  the  importance 
of  this  decision,  we  think  that  a  variety  of  task-clustering 
methods  should  be  available  to  the  user. 

2.  Enhancements  to  model  integration  capabilities.  The  OSBATS 
model  currently  includes  several  mechanisms  that  allow  the 
results  of  one  module  to  be  used  in  a  later  module. 

Additional  integration  of  the  modules  can  improve  their 
usefulness.  For  example,  there  is  a  need  to  incorporate  the 
simulation  requirements  determined  in  the  Simulation 
Configuration  Module  into  the  recommendations  of  the  other 
modules. 

3.  Expansion  of  the  model  to  new  training  technologies.  New 
options  that  are  available  to  the  training-system  designer 
should  be  evaluated  as  alternatives  to  traditional  simulation- 
based  methods.  Examples  of  training  methods  that  may  require 
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more  attention  include  embedded  training,  part-task  training 
and  skill  training. 

4.  Develop  ways  to  incorporate  school  requirements  and 

constraints  into  the  recommendations  of  the  model.  The  school 
may  require  that  certain  features  be  included  in  a  training- 
device  design.  Similarly,  the  school  may  have  constraints  on 
space  or  time  that  have  an  impact  on  the  optimal  training- 
device  concept.  There  is  a  need  for  methods  that  allow  all 
modules  to  consider  these  requirements  in  their  analysis. 

Data  Collection  Methods 


One  of  the  chief  barriers  to  the  application  of  the  OSBATS 
model  to  a  new  training  domain  is  the  effort  required  to  obtain 
the  necessary  data.  There  are  several  activities  that  could  be 
accomplished  to  reduce  the  effort  required  for  data  collection. 
First,  standard  procedures  for  data  collection  should  be 
developed.  To  the  extent  possible,  these  procedures  should 
minimize  the  requirement  for  judgments  by  subject-matter  experts. 
Where  precise  data  are  not  available,  methods  for  making 
assumptions  about  data  values  should  be  developed,  and  the  impact 
of  these  assumptions  on  the  results  of  the  model  determined. 
Finally,  procedures  should  be  developed  to  obtain  required  data 
from  existing  training  data  bases. 

Model  Calibration/Validation 

The  results  of  the  OSBATS  model  hinge  on  several  key 
assumptions  about  learning  and  transfer  processes.  Attempts  to 
validate  the  model  should  focus  on  these  key  assumptions.  The 
validation  process  will  involve  both  determining  the  best  value 
for  key  assumptions  (calibration)  and  testing  whether  this 
assumption  provides  an  adequate  account  of  the  learning  and 
transfer  processes  addressed  by  the  model  (validation) .  Because 
of  the  large  effort  required  for  model  validation,  we  recommend 
that  the  validation  effort  begin  with  a  careful  analytical 
evaluation  of  the  model  assumptions  to  determine  which 
assumptions  are  the  most  critical  to  the  model  results.  The 
sensitivity  analyses  conducted  under  the  current  effort  should 
provide  some  guidance  in  identifying  critical  assumptions. 

Software  Enhancements 


One  of  the  requirements  for  technology  transfer  will  be  the 
development  of  production-quality  software  representing  the 
OSBATS  model.  The  production  version  of  OSBATS  will  integrate 
analytic,  rule-based,  and  data  management  capabilities  of  the 
model.  We  expect  that  the  next  version  of  OSBATS  will 
incorporate  several  enhancements  to  the  model  software,  such  as  a 
simplified  user  interface  that  is  common  to  all  modules, 
increased  access  to  the  logic  that  is  used  in  rules  bases,  and 
access  to  the  data  that  form  the  basis  of  the  recommendations  of 
the  model.  In  addition,  the  next  version  of  the  software  should 
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incorporate  any  new  analytical  and  data  management  capabilities 
that  are  developed. 

Some  software  enhancements  may  be  investigated  using  the 
current  prototype  software.  Candidates  enhancements  for 
development  on  the  prototype  system  include  user  interface 
improvements,  more  sophisticated  help  capabilities,  and 
additional  displays.  Development  of  these  methods  on  the 
prototype  software  allows  these  methods  to  be  evaluated  before 
they  are  incorporated  into  the  production  software. 
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APPENDIX  A 


OSBATS  DATA  BASE 


APPENDIX  A — OSBATS  DATA  BASE 


I.  Task  Training  Requirements 
A.  Task  Learning  Points 


ABBREVIATED 


ATM  #  TASK  TITLE  ( 

1 . 

ENTRY 

STD 

1000 

1 

VFR  FLIGHT  ( 

0.6900 

0.70 

1003 

T/0  CHECKS  1 

0.4941 

0.70 

* 

1004 

HOVER  PWR  CK| 

0.5417 

0.70 

1005 

HOVER  FLT  | 

0.4941 

0.70 

« 

1006 

NORMAL  T/0  | 

0.4941 

0.70 

* 

1007 

S/MAX  P  T/O  1 

0.4941 

0.70 

* 

1008 

DECEL  ACCEL  | 

0.4941 

0.70 

* 

1009 

TRFC  PTRN  1 

0.4941 

0.70 

ICIO 

FUEL  CK  1 

0.6900 

0.70 

* 

1012 

DOPPLER  NAV  | 

0.0000 

0.70 

1013 

PRE  LNDG  CK  | 

0.6837 

0.70 

1014 

VMC  APPROACH! 

0 . 5000 

0.70 

« 

1015 

SHLW  APPR  1 

0.4583 

0.70 

« 

1016 

CONFIND  OPN  1 

0.5417 

0.70 

1017 

SLOPE  OPN  1 

0.4167 

0.70 

* 

1018 

TERR  FLT  T/0 I 

0.6250 

0.70 

* 

1019 

TERRAIN  FLT  j 

0.6250 

0.70 

1020 

HVR  GRND  CK  | 

0.6250 

0.70 

* 

1021 

NOE  DECEL  j 

0.5417 

0.70 

m 

1022 

TRN  FLT  APP  | 

0.5833 

0.70 

1023 

HI  SPD  FLT  1 

0.5417 

0.70 

* 

1024 

HVR_AUTOROT  ( 

0.5417 

0.70 

1025 

ENG  FAIL  HVRI 

0.5417 

0.70 

1026 

SIM  ENG  FAIL! 

0.5000 

0.70 

1027 

ENG  FAIL  SPDI 

0.4583 

0.70 

* 

1028 

MAN  THROTTLE! 

0.6250 

0.70 

* 

1029 

NO  SCAS/SAS  ! 

0.5000 

0.70 

1033 

ATT  RECVRY  | 

0.3000 

0.50 

1034 

RADIO  PROC  ! 

0 . 6833 

0.70 

1038 

IFR  RECVRY  ! 

0.3000 

0.50 

1039 

MSKG/UNMSKG  ! 

0.6917 

0.70 

1040 

TACT  COM  PRC! 

0.4941 

0.70 

- 

1041 

TRNSMT  RPT  ! 

0.5900 

0.60 

1042 

MOVE  TECH  ! 

0.4900 

0.50 

1043 

ACTN  CONTACT! 

0.4900 

0.50 

1044 

WIRE  OBSTACLI 

0.6900 

0.70 

1046 

AFTER_LNDG  ! 

0.6917 

0.70 

1048 

PIN/RDG  OPN  ! 

0.5417 

0.70 

1049 

FM  RAD  HOMNGI 

0.5833 

0.7 

1053 

TRNSMT  SIGNL! 

0.19 

0.2 

1054 

OP_RMS  ! 

0.0814 

0.7 

m 

1056 

WPNS  PROC  1 

0.0814 

0.7 

* 

1056 

OP_TURRET  ! 

0.0814 

0.7 

m 

1057 

OP  FFAR  ROC  1 

0.0833 

0.7 

« 

1058 

OP  TOW  MSL  1 

0.0814 

0.7 

1059 

ARM  SYS  MAL  ! 

0.0814 

0.7 

*  indicates  task  included  in  sample  problem 
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APPENDIX  A--OSBATS  DATA  BASE 


I .  Task  Training  Requirements 
A.  Task  Learning  Points 


ABBREVIATED 


ATM 

f  TASK  TITLE  | 

ENTRY 

STD 

1060 

SAFE  WEAP  I 

0.0814 

0.7 

* 

1061 

N0E_FIRIN6  1 

0.0824 

0.7 

1062 

FIRNG  POSTN  | 

0.0814 

0.7 

1063 

HI  SPD  DIVE  1 

0.3333 

0.7 

* 

2000 

STD_AUT0R0T  | 

0.5417 

0.7 

2001 

LOW  AUTOROT  1 

0.5417 

0.7 

2002 

HYDR  SYS  MALi 

0.4583 

0.7 

« 

2003 

S/A_TRQ  MAL  | 

0.3333 

0.7 

« 

2004 

L-A,H-S  AR  1 

0.3333 

0.7 

2005 

LO-SPD  AUTORI 

0.4167 

0.7 

3000 

EMER  START  | 

0.407 

0.7 

3004 

CLUTCH  FL  1 

0.4167 

0.7 

3005 

ENG_RESTART  | 

0 . 4268 

0.7 

3006 

ENG_OVERSPD  | 

0.4167 

0.7 

3007 

ENG  UNDERSPDI 

0.4167 

0.7 

3006 

DROOP  CMP  FLI 

0.4167 

0.7 

3009 

ENG  CMPR  STLI 

0.4167 

0.7 

♦ 

3010 

TAIL  ROT  FL  I 

0.3333 

0.7 

* 

3011 

MAST  BUMPING! 

0.6833 

0.7 

3012 

ENG  FIR  STRTi 

0.3333 

0.7 

3013 

FUSL  FIR  GUN! 

0.3333 

0.7 

3014 

FUSL  FIR  GRDI 

0.6837 

0.7 

3015 

FUSL  FIR  FLTI 

0.5833 

0.7 

3016 

ENG  FIR  FLT  I 

0 . 5833 

0.7 

3017 

ENG  FR  L/ALTI 

0.5833 

0.7 

3018 

ENG  FR  CRUISI 

0.5833 

0.7 

3019 

ELEC  FIR  FLTI 

0.5833 

0.7 

3020 

ELEC  FIR  GRDI 

0.5833 

0.7 

3021 

WING_STORES  j 

0.5833 

0.7 

3022 

ECS_FUMES  1 

0.6833 

0.7 

3023 

FUME  ELIMIN  | 

0.6667 

0.7 

3024 

FUEL  BST  FL  I 

0.6667 

0.7 

3025 

THROTTLE  PL  j 

0.5833 

0.7 

3026 

ELECTRIC  FL  j 

0.5833 

0.7 

3027 

OVRHTD  BTTRYI 

0.6833 

0.7 

3028 

CAUTN  LGHT  j 

0.6833 

0.7 

3029 

EMER_DESCENT  j 

0.5833 

0.7 

4t 

3030 

LNDG/TREES  j 

0.5833 

0.7 

3031 

DTCH  PWR  ON  1 

0.5833 

0.7 

3032 

DTCH  PWR  OFF! 

0.5833 

0.7 

« 

3033 

SCAS_FAILURE| 

0.5 

0.7 

3034 

BAILOUT  PRC  1 

0.5 

0.7 

3035 

WNG  STR  JTSNI 

0.5833 

0.7 

3036 

TOW  EMER  PRCI 

0.4167 

0.7 

3037 

RUNAWAY  GUN  | 

0.4167 

0.7 

4003 

ZERO  OIL  PR  1 

0.6833 

0.7 

4004 

HI  EN  OIL  PRI 

0 . 6833 

0.7 

4009 

SCAS  PTCH  HOI 

0.5 

0.7 
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I.  Task  Training  Requirements 
A.  Task  Learning  Points 

ABBREVIATED  | 

ATM  #  TASK  TITLE  I  ENTRY  STD 


4010  SCAS_ROLL_HO I  0.5  0.7 

4011  SCAS_PTCH_MT I  0.5  0.7 

4012  SCAS_ROLL_MT I  0.5  0.7 

4013  SCAS_YAW_MT  |  0.5  0.7 

4017  T/R_L0SS  |  0.29  0.3 

4018  T/R_THRST_LS I  0.29  0.3 

4019  T/R_PTCH_FRZ I  0.2  0.5 

4020  GEAR_BX_LOSS I  0.29  0.3 

4022  FLAME_OUT  |  0.5  0.7 

4028  ENG_CHP_LGHT I  0.6753  0.7 

4030  HI_TR_0IL_PRi  0.6753  0.7 

4031  HI_ENG_RPM  |  0.4167  0.7 

4032  LC_ENG_RPM  |  0.4167  0.7 

4035  CL0S_INL_GDV|  0.5833  0.7 

4036  0PEN_INL_GDV I  0.69  0.7 

4037  ENG_0IL_L0SS|  0.6833  0.7 

4038  TR_OIL_LOSS  |  0.6833  0.7 

4039  TR_0IL_BYPAS|  0.6833  0.7 

4040  TR_CHIP_LGHT I  0.6833  0.7 

4042  ENG_DRV  FUEL |  0.6833  0.7 

4045  FORW_FUEL_LT I  0.1628  0.7 

4046  AFT_FUEL_LT  |  0.1628  0.7 

4047  FUEL_FLTR_LT I  0.6833  0.7 

4048  L0_FUEL_LT  |  0.6833  0.7 

4049  HyD/SyS_l_FL|  0.0833  0.7 

4050  HyD/SYS_2_FL I  0.0833  0.7 

4051  DUAL_HYDR_FL I  0  0.4 

4052  CMPLT_ELC_FL I  0.5833  0.7 

4055  D/C_GNRTR_FLj  0.3333  0.7 

4056  ALTERNATR_FL I  0.3333  0.7 

4057  A/C  INVRT  FL  0.4167  0.7 


A-3 


APPENDIX  A — OSBATS  DATA  BASE 


I.  Task  Training  Requirements 

B.  Task  Simulation  Evaluation  Factors 


ATM 

ABBREVIATED  | 
#  TASK  TITLE  \ 

ABS 

RQRD 

SPEC 

wx 

SPEC 

SIT 

SPEC 

EQUIP 

TRNG 

EFFECT 

TIME 

SAVE 

1000 

VFR_FLIGHT  | 

0 

0 

0 

0 

0 

0 

♦ 

1003 

T/0_CHECKS  1 

0 

0 

0 

0 

0 

1 

1004 

HOVER  PWR  CK  | 

0 

0 

0 

0 

0 

1 

1005 

H0VER_FLT  | 

0 

1 

0 

0 

0 

0 

1006 

NORMAL  T/0  ( 

0 

1 

0 

0 

0 

1 

« 

1007 

S/MAX  P  T/0  1 

0 

1 

0 

0 

0 

1 

1008 

DECEL_ACCEL  | 

0 

1 

0 

0 

0 

0 

* 

1009 

TRFC  PTRN  1 

0 

1 

0 

0 

0 

0 

1010 

FUEL  CK  1 

0 

0 

0 

0 

0 

0 

« 

1012 

DOPPLER_NAV  ( 

0 

0 

0 

0 

0 

0 

♦ 

1013 

PRE  LNDG  CK  I 

0 

0 

0 

0 

0 

0 

1014 

VMC  APPROACH  | 

0 

1 

0 

0 

0 

1 

1015 

SHLW  APPR  1 

0 

1 

0 

0 

0 

1 

» 

1016 

C0NFIND_0PN  1 

0 

1 

0 

0 

0 

1 

1017 

SL0PE_0PN  1 

0 

1 

0 

0 

0 

1 

* 

1018 

TERR  FLT  T/0  1 

0 

1 

0 

0 

0 

1 

1019 

TERRAIN  FLT  | 

0 

1 

0 

0 

0 

1 

1020 

HVR  GRND  CK  I 

0 

1 

0 

0 

0 

0 

« 

1021 

NOE  DECEL  | 

0 

1 

0 

0 

0 

0 

* 

1022 

TRN  FLT  APP  | 

0 

1 

0 

0 

0 

1 

1023 

HI  SPD  FLT  1 

0 

0 

0 

0 

0 

0 

« 

1024 

HVR  AUTOROT  | 

0 

0 

.  0 

0 

0 

1 

1025 

ENG  FAIL  HVR  | 

0 

1 

0 

0 

0 

1 

« 

1026 

SIM  ENG  FAIL  | 

0 

1 

0 

0 

0 

1 

1027 

ENG  FAIL  SPD  | 

0 

1 

0 

0 

0 

1 

» 

1028 

MAN_THROTTLE  | 

0 

1 

0 

0 

0 

1 

* 

1029 

NO  SCAS/SAS  1 

0 

1 

0 

0 

0 

0 

1033 

ATT  RECVRY  | 

0 

1 

0 

0 

0 

1 

1034 

RADIO_PROC  1 

0 

0 

0 

0 

0 

0 

1038 

IFR_RECVRY  | 

0 

1 

0 

0 

0 

1 

1039 

MSKG/UNMSKG  | 

0 

1 

0 

0 

0 

1 

1040 

TACT  COM  PRC  | 

0 

0 

1 

0 

0 

1 

1041 

TRNSMT  RPT  | 

0 

0 

1 

0 

0 

1 

1042 

MOVE  TECH  ( 

0 

1 

1 

0 

0 

1 

1043 

ACTN  CONTACT  | 

0 

1 

1 

0 

0 

1 

1044 

WIRE  OBSTACL  1 

0 

0 

0 

1 

0 

1 

1045 

AFTER  LNDG  | 

0 

0 

0 

0 

0 

0 

« 

1048 

PIN/RDG_OPN  1 

0 

1 

0 

1 

0 

1 

1049 

FM  RAD  HOMNG  | 

0 

1 

0 

0 

0 

1 

1053 

TRNSMT  SIGNL  | 

0 

0 

0 

0 

0 

0 

1054 

OP  RMS  1 

0 

1 

1 

1 

1 

1 

1055 

WPNS  PROC  1 

0 

0 

1 

1 

1 

1 

1056 

OP_TURRET  1 

0 

1 

1 

1 

1 

1 

» 

1057 

OP  FFAR  ROC  ( 

0 

1 

1 

1 

1 

1 

1058 

OP  TOW  MSL  1 

0 

1 

1 

1 

1 

1 

1059 

ARM  SYS  MAL  I 

1 

0 

0 

0 

0 

0 

1060 

SAFE  WEAP  1 

0 

0 

0 

0 

0 

0 

« 

1061 

NOE_FIRING  1 

0 

1 

1 

1 

1 

1 
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APPENDIX  A--OSBATS  DATA  BASE 


I .  Task  Training  Requirements 

B.  Task  Simulation  Evaluation  Factors 


ATM 

ABBREVIATED  | 

#  TASK  TITLE  | 

ABS 

RQRD 

SPEC 

WX 

SPEC 

SIT 

SPEC 

EQUIP 

TRNG 

EFFECT 

TIME 

SAVE 

1062 

FIRNG  POSTN  | 

0 

1 

1 

1 

1 

1 

1063 

HI  SPD  DIVE  1 

0 

1 

0 

0 

0 

0 

*  2000 

STD_AUT0R0T  | 

0 

1 

0 

0 

1 

1 

2001 

L0W_AUT0R0T  1 

0 

1 

0 

0 

1 

1 

2002 

HYDR  SYS  MAL  | 

0 

1 

0 

0 

1 

1 

*  2003 

S/A  TRQ  MAL  | 

0 

1 

0 

0 

1 

1 

*  2004 

L-A,H-S_AR  ( 

0 

1 

0 

0 

1 

1 

2005 

LO-SPD  AUTOR  | 

0 

1 

0 

0 

1 

1 

3000 

EMER_START  | 

1 

0 

0 

0 

0 

0 

3004 

CLUTCH  FL  1 

1 

0 

0 

0 

0 

0 

3005 

ENG_RESTART  | 

1 

0 

0 

0 

0 

0 

3006 

ENG  OVERSPD  | 

1 

0 

0 

0 

0 

0 

3007 

ENG_UNDERSPD  | 

1 

0 

0 

0 

0 

0 

♦  3008 

DROOP  CMP  FL  1 

1 

0 

0 

0 

0 

0 

3009 

ENG  CMPR  STL  I 

1 

0 

0 

0 

0 

0 

*  3010 

TAIL  ROT  FL  | 

1 

0 

0 

0 

0 

0 

•  3011 

MAST  BUMPING  | 

1 

0 

0 

0 

0 

0 

3012 

ENG  FIR  STRT  | 

1 

0 

0 

0 

0 

0 

3013 

FUSL  FIR  GUN  | 

1 

0 

0 

0 

0 

0 

3014 

FUSL  FIR  GRD  | 

1 

0 

0 

0 

0 

0 

3015 

FUSL  FIR  FLT  I 

1 

0 

0 

0 

0 

0 

3016 

ENG  FIR  FLT  | 

1 

0 

0 

0 

0 

0 

3017 

ENG  FR  L/ALT  I 

1 

0 

0 

0 

0 

0 

3018 

ENG  FR  CRUIS  I 

1 

0 

0 

0 

0 

0 

3019 

ELEC  FIR  FLT  I 

1 

0 

0 

0 

0 

0 

3020 

ELEC  FIR  GRD  I 

1 

0 

0 

0 

0 

0 

3021 

WING  STORES  | 

1 

0 

0 

0 

0 

0 

3022 

ECS  FUMES  1 

0 

0 

0 

0 

0 

0 

3023 

FUME  ELIMIN  | 

0 

0 

0 

0 

0 

0 

3024 

FUEL  BST  FL  | 

1 

0 

0 

0 

0 

0 

*  3025 

THROTTLE  FL  | 

1 

0 

0 

0 

0 

0 

3026 

ELECTRIC  FL  j 

1 

0 

0 

0 

0 

0 

3027 

OVRHTD_BTTRY  1 

1 

0 

0 

0 

0 

0 

3028 

CAUTN_LGHT  | 

1 

0 

0 

0 

0 

0 

3029 

EMER  DESCENT  | 

1 

0 

0 

0 

0 

0 

*  3030 

LNDG/TREES  | 

1 

0 

0 

0 

0 

0 

3031 

DTCH  PWR  ON  1 

1 

0 

0 

0 

0 

0 

3032 

DTCH  PWR  OFF  I 

1 

0 

0 

0 

0 

0 

3033 

SCAS  FAILURE  | 

0 

0 

0 

0 

1 

1 

3034 

BAILOUT  PRC  I 

1 

0 

0 

0 

0 

0 

3035 

WNG  STR  JTSN  I 

1 

0 

0 

0 

0 

0 

3036 

TOW  EMER  PRC  | 

1 

0 

0 

0 

0 

0 

3037 

RUNAWAY_GUN  | 

1 

0 

0 

0 

0 

0 

4003 

ZERO  OIL  PR  1 

1 

0 

0 

0 

0 

0 

4004 

HI  EN  OIL  PR  1 

1 

0 

0 

0 

0 

0 

4009 

SCAS  PTCH  HO  1 

1 

0 

0 

0 

0 

0 

4010 

SCAS  ROLL  HO  | 

1 

0 

0 

0 

0 

0 

4011 

SCAS  PTCH  MT  | 

1 

0 

0 

0 

0 

0 
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I  . 


ATM 


4012 

4013 

4017 

4018 

4019 

4020 
4022 
4028 

4030 

4031 

4032 

4035 

4036 

4037 

4038 

4039 

4040 
4042 

4045 

4046 

4047 

4048 

4049 

4050 

4051 

4052 

4055 

4056 

4057 


Task  Training  Requirements 

B.  Task  Simulation  Evaluation  Factors 


ABBREVIATED 
#  TASK  TITLE 


ABS 

RQRD 


SCAS_ROLL_MT  |  1 
SCAS_YAW_MT  |  1 
T/R_L0SS  I  1 
T/R_THRST_LS  |  1 
T/R_PTCH_FRZ  |  0 
GEAR_BX_LOSS  |  1 
FLAME_0UT  |  1 
ENG_CHP_LGHT  |  1 
HI_TR_OIL_PR  I  1 
HI_ENG_RPM  (  1 
L0_ENG_RPM  I  1 
CLOS_rNL_GDV  |  1 
0PEN_INL_GDV  (  1 
ENG_0IL_L0SS  |  1 
TR_OIL_LOSS  I  1 
TR_OIL_BYPAS  |  1 
TR_CHIP_LGHT  |  1 
ENG_DRV_FUEL  (  1 
FORW_FUEL_LT  j  1 
AFT_FUEL_LT  |  1 
FUEL_FLTR_LT  |  ' 1 
L0_FUEL_LT  |  1 
HYD/SYS_1_FL  |  1 
HYD/SYS_2_FL  |  1 
DUAL_HYDR_FL  |  1 
CMPLT_ELC_FL  |  1 
D/C_GNRTR_FL  |  1 
ALTERNATR_FL  (  1 
A/C  INVRT  FL  |  1 


SPEC  SPEC  SPEC  TRNG  TIME 

WX  SIT  EQUIP  EFFECT  SAVE 


0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 


0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 


0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 


0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 


1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 
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II.  Other  Task  Data 

A.  Task  Training  Hours  and  Costs 


OTHER 


ATM  # 

ABBREVIATED  ( 
TASK  TITLE  | 

CLASSRM  -  EQUIPMENT  HOURS  - 

HOURS  NON-FLT  FLT  SETUP 

EQUIP 

($000) 

1000 

VFR  FLIGHT  | 

0.1 

0.00 

0.00 

0 

0 

1003 

T/0  CHECKS  1 

2 

0.70 

0 . 00 

0 

0 

1004 

HOVER  PWR  CK| 

0.2 

0.00 

0.20 

0 

0 

« 

1005 

HOVER  FLT  | 

0 

0.00 

0.40 

0 

0 

* 

1006 

NORMAL  T/0  | 

0 

0.10 

0.40 

0.1 

0 

« 

1007 

S/MAX  P  T/0  1 

0 

0.10 

0 . 40 

0.1 

0 

* 

1008 

DECEL  ACCEL  \ 

0 

0.00 

0.60 

0 

0 

« 

1009 

TRFC  PTRN  1 

0 

0 . 10 

0.50 

0 

0 

1010 

FUEL  CK  1 

0 

0.10 

0.20 

0 

0 

« 

1012 

DOPPLER  NAV  ( 

6 

0.30 

0 . 70 

0 

0 

* 

1013 

PRE  LNDG  CK  | 

0 . 2 

0.00 

0.10 

0 

0 

* 

1014 

VMC  APPROACH  1 

0 

0. 10 

0.60 

0. 1 

0 

* 

1015 

SHLW  APPR  1 

0 

0.10 

0.60 

0.1 

0 

« 

1016 

CONFIND  OPN  1 

0 

0.10 

0.60 

0.1 

0 

1017 

SLOPE  OPN  1 

0 

0.10 

0.30 

0 

0 

m 

1018 

TERR  FLT  T/0| 

0 

0.00 

0.10 

0 

0 

« 

1019 

TERRAIN_FLT  \ 

0 

0.00 

0.40 

0 

0 

1020 

HVR  GRND  CK  | 

0.2 

0.10 

0.40 

0 

0 

« 

1021 

N0E_DECEL  '  | 

0 

0.00 

0.30 

0 

0 

« 

1022 

TRN  FLT  APP  | 

0 

0.00 

0.20 

0 

0 

1023 

HI  SPD  FLT  1 

0.1 

0.10 

0.10 

0 

0 

* 

1024 

HVR_AUT0R0T  | 

0 

0.10 

0.30 

0 . 1 

0 

1025 

ENG  FAIL  HVRI 

0 

0.10 

0.30 

0 

0 

« 

1026 

SIM  ENG  FAILI 

0 

0.10 

0.20 

0 

0 

1027 

ENG  FAIL  SPDI 

0 

0.10 

0.10 

0 

0 

1028 

MAN  THROTTLE  1 

0 . 5 

0. 10 

0.60 

0. 1 

0 

« 

1029 

WO  SCAS/SAS  1 

0. 1 

0.10 

0.50 

0 

0 

1033 

ATT  RECVRY  | 

0 

0 . 20 

0 . 10 

0 

0 

1034 

RADIO  PROC  1 

0 

0.20 

0.10 

0 

0 

1038 

IFR  RECVRY  | 

0 

0. 20 

0 . 30 

0 . 1 

0 

1039 

MSKG/UNMSKG  | 

0 

o.oc 

0.40 

0. 1 

0 

1040 

TACT  COM  PRCI 

0 . 2 

0.00 

0.00 

0 

0 

1041 

TRNSMT  RPT  | 

0.2 

0.00 

0.00 

0 

0 

« 

1042 

MOVE_TECH  1 

0.4 

0.30 

0.00 

0 

0 

1043 

ACTN_CONTACT 1 

0.3 

0.00 

0.00 

0 

0 

1044 

WIRE  OBSTACLI 

0 

0.10 

0.00 

0 

0 

1045 

AFTER  LNDG  ( 

0 . 1 

0.10 

0.10 

u 

0 

* 

1048 

P^N/RDG  OPN  1 

0.3 

0.00 

0.60 

0 . 1 

0 

1049 

FM  RAD  HOMNGI 

0.3 

0.1 

0.3 

0.1 

0 

1053 

TRNSMT  SIGNLI 

0 . 3 

0 . 1 

0 

0 

0 

1054 

OF  RMS  1 

5 

0.2 

1 . 5 

0 . 5 

1 . 39 

1055 

WPNS  PROC  1 

3 

0.3 

2 

0 . 6 

1  87 

1056 

OP_TURRET  1 

4 

0. 1 

2 

0.6 

0 . 3 

« 

1057 

OP  FFAR  ROC  1 

0 . 1 

0 . 1 

2 

0 . 6 

4 .98 

* 

1058 

OP  TOW  MSL  1 

9 

0.3 

2 

0.6 

0.5 

1059 

ARM  S/S  MAL  | 

0 . 5 

0 . 3 

0 

0 

0 

1060 

SAFE  WEAP  1 

0.3 

0.3 

0 . 5 

0 

0 

* 

1061 

NOE  FIRING  1 

0 . 2 

0 . 3 

3 . 5 

0 . 6 

8 . 87 
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APPENDIX  A — OSBATS  DATA  BASE 


II.  Other  Task  Data 

A.  Task  Training  Hours  and  Costs 


ABBREVIATED  | 
ATM  #  TASK  TITLE  | 

CLASSRM 

HOURS 

-  EQUIPMENT  HOURS  - 

NON-FLT  FLT  SETUP 

OTHER 
EQUIP 
( $000) 

1062 

1 

FIRNG  POSTN  | 

0 

0.4 

0.5 

0 

0 

1063 

HI  SPD  DIVE  1 

0 

0.2 

0.5 

0 . 1 

0 

« 

2000 

STD  AUTOROT  | 

0 

0.1 

0.7 

0.2 

0 

2001 

LOW  AUTOROT  | 

0 

0.2 

0.7 

0.2 

0 

2002 

HYDR  SYS  MALI 

0 

0.2 

0.7 

0.2 

0 

♦ 

2003 

S/A  TRQ  MAL  | 

0 

0.1 

0.7 

0.2 

0 

2004 

L-A,H-S  AR  1 

0 

0.1 

0.7 

0 . 2 

0 

2005 

LO-SPD  AUTORl 

0 

0.1- 

0.7 

0.2 

0 

3000 

EMER_START  | 

0.2 

0.1 

0 

0 

0 

3004 

CLUTCH  FL  1 

0.3 

0.1 

0 

0 

0 

3005 

ENG  RESTART  | 

0.2 

0 . 1 

0 

0 

0 

3006 

ENG_OVERSPD  | 

0 . 2 

0 . 1 

0 

0 

0 

3007 

ENG  UNDERSPDI 

0.2 

0.1 

0 

0 

0 

* 

3008 

DROOP  CMP  FLI 

0.2 

0,3 

0 

0 

0 

3009 

ENG  CMPR  STLI 

0.3 

0.1 

0 

0 

0 

3010 

TAIL  ROT  FL  I 

0 

0.1 

0 

0 

0 

m 

3011 

MAST_BUMPING | 

0.2 

0.1 

0 

0 

0 

30^  ^ 

ENG  FIR  STRTI 

0.2 

0. 1 

0 

0 

0 

3013 

FUSL  FIR  GUNI 

0.1 

0.1 

0 

0 

0 

3014 

FUSL  FIR  GRDI 

0.1 

0.1 

0 

0 

0 

-  :  5 

FUSL  FIR  FLTl 

0.1 

0.1 

0 

0 

0 

30x6 

ENG  FIR  FLT  I 

0.1 

0.1 

0 

0 

0 

3017 

ENG  FR  L/ALTt 

0.1 

0.1 

0 

0 

0 

3018 

ENG  FR  CRUISI 

0.1 

0.1 

0 

0 

0 

3019 

ELEC  FIR  FLTl 

0.2 

0. 1 

0 

0 

0 

3020 

ELEC  FIR  GRDI 

0.1 

0.1 

0 

0 

0 

3021 

WING  STORES  | 

0.2 

0.1 

0 

0 

0 

3022 

ECS  FUMES  1 

0.2 

0. 1 

0 

0 

0 

3023 

FUME  ELIMIN  | 

0.2 

0, 1 

0 

0 

0 

3024 

FUEL  BST  FL  I 

0.3 

0.2 

0 

0 

0 

3025 

THROTTLE_FL  | 

0.1 

0.1 

0 

0 

0 

3026 

ELECTRIC_FL  | 

0.1 

0.1 

0 

0 

0 

3027 

OVRHTD  BTTRYI 

0.2 

0.  1 

0 

0 

0 

3028 

CAUTN  LGHT  ( 

0.1 

0.1 

0 

0 

0 

3029 

EMER_DESCENT| 

0 

0.1 

0 

0 

0 

« 

3030 

LNDG/TREES  ( 

0 

0.2 

0 

0 

0 

3031 

DTCH  PWR  ON  1 

0 

0.2 

0 

0 

0 

3032 

DTCH  PWR  OFFI 

0 

0.2 

0 

0 

0 

« 

3033 

SCAS  FAILURE  1 

1.2 

0.1 

0.2 

0 

0 

3034 

BAILOUT  PRC  1 

0 

0 . 2 

0 

0 

0 

3035 

WNG  STR  JTSNI 

0.2 

0.1 

0 

0 

0 

3036 

TOW  EMER  PRCI 

0 . 3 

0.3 

0 

0 

0 

3037 

RUNAWAY  GUN  | 

0.2 

0.2 

0 

0 

0 

4003 

ZERO  OIL  PR  1 

0 . 1 

0, 1 

0 

0 

0 

4004 

HI  EN  OIL  PRI 

0.1 

0.1 

0 

0 

0 

4009 

SCAS  PTCH  HOI 

0 . 2 

0.2 

0 

0 

0 

4010 

SCAS  ROLL  HOI 

0.2 

0.2 

0 

0 

0 

4011 

SCAS  PTCH  MT| 

0.2 

0,2 

0 

0 

0 
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APPENDIX  A — OSBATS  DATA  BASE 


II.  Other  Task  Data 

A.  Task  Training  Hours  and  Costs 


ABBREVIATED 
ATM  #  TASK  TITLE 


4012  SCAS_ROLL_MT 

4013  SCAS_YAW_MT 

4017  T/R_L0SS 

4018  T/R_THRST_LS 

4019  T/R_PTCH_PRZ 

4020  GEAR_BX_LOSS 
4022  FLAME_0UT 
4028  ENG_CHP_LGHT 

4030  HI_TR_OIL_PR 

4031  HI_ENG_RPM 

4032  L0_ENG_RPM 

4035  CL0S_INL_GDV 

4036  OPEN_INL_GDV 

4037  ENG_0IL_L0SS 

4038  TR_0IL_L0SS 

4039  TR_OIL_BYPAS 

4040  TR_CHIP_LGHT 
4042  ENG_DRV_FUEL 

4045  FORW_FUEL_LT 

4046  AFT_FUEL_LT 

4047  FUEL_FLTR_LT 

4048  L0_FUEL_LT 

4049  HYD/SYS_1_FL 

4050  HYD/SYS_2_FL 

4051  DUAL_HYDR_FL 

4052  CMPLT_ELC_FL 

4055  D/C_GNRTR_FL 

4056  ALTERNATR_PL 

4057  A/C  INVRT  FL 


OTHER 

CLASSRM  -  EQUIPMENT  HOURS  -  EQUIP 


HOURS 

NON-FLT 

FLT 

SETUP 

($000) 

0.2 

0.2 

0 

0 

0 

0.2 

0.2 

0 

0 

0 

0.2 

0.1 

0 

0 

0 

0.2 

0.1 

0 

0 

0 

0.2 

0.1 

0 

0 

0 

0.2 

0.2 

0 

0 

0 

0.2 

0.1 

0 

0 

0 

0.2 

0.1 

0 

0 

0 

0.1 

0.1 

0 

0 

0 

0.2 

0.1 

0 

0 

0 

0.2 

0.1 

0 

0 

0 

0.2 

0.1 

0 

0 

0 

0.2 

0.1 

0 

0 

0 

0.1 

0.1 

0 

0 

0 

0.1 

0.1 

0 

0 

0 

0.2 

0.1 

0 

0 

0 

0.2 

0.1 

0 

0 

0 

0  3 

0.1 

0 

0 

0 

0.2 

0.1 

0 

0 

0 

0.2 

0.1 

0 

0 

0 

0.2 

0.1 

0 

0 

0 

0.2 

0.1 

0 

0 

0 

0.5 

0.2 

0.3 

0.1 

0 

0.5 

0.2 

0.3 

0 . 1 

■  0 

0.5 

0.2 

0.3 

0.1 

0 

0.2 

0.1 

0 

0 

0 

0.2 

0.1 

0 

0 

0 

0.2 

0.1 

0 

0 

0 

0.2 

0.1 

0 

0 

0 
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APPENDIX  A — OSBATS  DATA  BASE 


III.  Training  Device  Data 

A.  Training  Device  Costs 


COST  (K) -  LIFE-  UTIL 


MEDIUM  1 

INVEST 

FIXED/YR 

VAR/HR 

CYCLE 

(HR/YR) 

CLASSROOM  1 

0.0 

5.0 

0.0299 

50 

2000 

COCKPIT  PROC  TRNR| 

80.0 

2.0 

0.0500 

10 

2000 

DOPPLER  NAV  TRNR  | 

40.0 

1.0 

0,0305 

10 

2000 

IMC  PLIGHT  TRNR  ( 

3600.0 

8.0 

0.0700 

10 

3000 

VMC  PLIGHT  TRNR  | 

8000.0 

40.0 

0.0800 

10 

3000 

WEAPONS  TRNR  | 

640.0 

8.0 

0.0500 

10 

2000 

NOE  NAV  TRNR  | 

10.0 

0.4 

0.0300 

10 

2000 

ACT  EQUIP ( HLCPTR ) 1 

5000.0 

100.0 

0.6250 

10 

2500 
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APPENDIX  A-0S8ATS  DATA  BASE 


II!.  Training  Device  Data 

B.  Training  Device  Cue  and  Response  Capabilities 


1 

1 

MEDIUM  1 

RES 

CONT 

VISUAL 

TXTR 

FRONT 

FOV 

SIDE 

FOV 

F/X 

PNTS 

. -MOT 

F/X 

AREA  PLTFM 

ION- 

SEAT 

SOUND 

F/X 

MAP 

AREA 

CLASSROOM  1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

COCKPIT  PROC  TRNRI 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0.27 

0 

DOPPLER  NAV  TRNR  | 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0.27 

0 

IHC  FLIGHT  TRNR  | 

0 

0 

0 

0 

0 

0 

0 

0.63 

0.36 

0.27 

0.58 

VMC  FLIGHT  TRNR  | 

0.8 

0.58 

0.5 

0.86 

0.51 

0.18 

0.45 

0.9 

0.36 

0.54 

0.62 

WEAPONS  TRNR  | 

0.93 

0.9 

0.9 

0.77 

0 

0.9 

0.45 

0 

0 

0.54 

0.82 

NOE  NAV  TRNR  j 

0.93 

0.9 

0.9 

0.86 

0.51 

0 

0 

0 

0 

0.27 

0.62 

ACTUAL  EQUIPMENT  | 

1 

1 

1 

t 

1 

1 

1 

1 

1 

1 

1 
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APPENDIX  A-0S8ATS  DATA  BASE 


HI.  Trafnfng  Device  Data 

C.  Training  Device  Instructional  Features 


•MEDIUM 

PERF 

MEAS 

PEPP 

INO 

PROC 

MNTR 

PERF 

ALRT 

AUG 

FOBK 

AUG 

CUES 

RECRO  SYSTM 
PLBK  FREEZ 

GRPH 

RPLY 

INIT 

CONO 

SCEN 

CNTL 

CRSH 

OVRD 

RE¬ 

SET 

PARM 

FREEZ 

CLASSROOM 

0 

n 

C 

n 

n 

1 

0 

0 

0 

1 

0 

0 

0 

0 

COCKPIT  PROC  TRNR 

1 

0 

0 

0 

0 

0 

0 

0 

0 

1 

0 

0 

0 

0 

DOPPLER  NAV  TRNR 

1 

0 

1 

0 

0 

0 

0 

0 

0 

1 

1 

0 

0 

0 

IMC  FLIGHT  TRNR 

1 

0 

1 

0 

0 

0 

1 

1 

0 

1 

1 

1 

1 

0 

VMC  FLIGHT  TRNR 

1 

0 

1 

0 

1 

1 

1 

1 

0 

1 

1 

1 

1 

1 

WEAPONS  TRNR 

1 

0 

0 

0 

1 

0 

0 

0 

0 

I 

1 

0 

0 

0 

NOE  NAV  TRNR 

1 

1 

0 

•0 

0 

1 

0 

0 

0 

1 

1 

0 

0 

0 

ACTUAL  EQUIPMENT 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 
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HI.  Training  Oevic 
C.  Training 


1  PLT 

PSTN 

REAL 

AUTO 

AOJN 

AOPT 

CUE 

MEDIUM 

IFREEZ 

FREE! 

TIME 

DEMO 

CAI 

TRN6 

COACH 

CLASSROOM 

1  0 

0 

0 

0 

0 

0 

0 

uOCKPii  PtiOL  IRNR 

1  0 

G 

0 

A 

ft 

w 

0 

ft 

DOPPLER  NAV  TRNR 

1  0 

0 

0 

0 

0 

0 

0 

IMC  FLIGHT  TRNR 

1  0 

0 

0 

1 

0 

0 

0 

VMC  FLIGHT  TRNR 

1  1 

1 

1 

1 

0 

0 

0 

WEAPONS  TRNR 

1  0 

0 

0 

1 

0 

0 

0 

NOE  NAV  TRNR 

1  0 

0 

0 

0 

0 

0 

0 

ACTUAL  EQUIPMENT 

1  0 

0 

0 

0 

0 

0 

0 
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IV.  Fidelity 
A 

DIMENSION 
VIS  RESOLTN 

VIS  CONTENT 

VIS  TEXTURE 

FRONT  FOV 

SIDE  FOV 

SPL  F/X  PNTS 

SPL  F/X  AREA 

MTN  PLTFRM 

MTN  SEAT 

AUDIO  F/X 


Dimension  Data 

Fidelity  Dimensions  and  Levels 

TECH 


LEVEL 

NAME 

PERF 

1 

M2  §  0.3  KM 

0.52 

2 

M2  @  0.5  KM 

0.71 

3 

M2  @  1.0  KM 

0.86 

4 

M2  @  2.0  KM 

0.93 

5 

M2  @  3.0  KM 

0.95 

6 

M2  @  4.0  KM 

0.96 

1 

PLANE  WITH  TREES 

0.5 

2 

ADD  GENERIC  FEATRS 

0.58 

3 

REALISTIC  DENSITY 

0.66 

4 

LOW  DNSTY  HYDROGRAPH 

0.74 

5 

MED  DNSTY  HYDROGRAPH 

0.82 

6 

HI  DNSTY  HYDROGRAPH 

0.9 

1 

LINES  +  POLYGONS 

0.5 

2 

MODULATING  FNCTNS 

0.62 

3 

FEW  DIGITIZED  FOTOS 

0.74 

4 

MORE  DIGITIZED  FOTOS 

0 . 84 

5 

MANY  DIGITIZED  FOTOS 

0.9 

1 

40  X  40  DEG 

0.77 

2 

40  X  50  DEG 

0.86 

3 

40  X  60  DEG 

0.94 

1 

1  -  40  X  40  DEG 

0.46 

2 

1  -  40  X  50  DEG 

0.51 

3 

1  -  50  X  50  DEG 

0.57 

4 

1  -  50  X  60  DEG 

0.62 

5 

2  -  40  X  50  DEG 

0.72 

6 

2  -  40  X  60  DEG 

0.79 

7 

2  -  50  X  60  DEG 

0.88 

1 

NONE 

0 

2 

CULTURAL  LGHTS 

0.09 

3 

ADD  WPNS  BLAST 

0.18 

4 

ADD  DAMGD  VEHICLES 

0.36 

5 

ADD  AIRBRN  VEHICLES 

0 . 63 

6 

ADD  MVNG  GND  VEHCLE 

0.9 

1 

NONE 

0 

2 

SMOKE  AND  DUST 

0.45 

3 

ROTOR  WASH 

0.9 

1 

NONE 

0 

2 

3  D.F. 

0.36 

3 

5  D.F. 

0.63 

4 

6  D.F. 

0.9 

1 

STATIONARY 

0 

2 

SEAT  SHAKER 

0.36 

3 

ADD  G-SEAT 

0.9 

1 

NONE 

0 

2 

WPNS , SKID , SOME  FAILS 

0.27 

3 

ADD  NORMAL  OPS  NOISE 

0.54 

4 

ADD  ABNORMAL  OPS  NSE 

0.9 
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IV.  Fidelity  Dimension  Data 

A.  Fidelity  Dimensions  and  Levels 


1 

DIMENSION  (  LEVEL 

NAME 

TECH 

PERF 

_  —  -  -  ,  - 
DATA  SZ/CAL  | 

1 

5  X  5  KM 

0.5 

1 

2 

10  X  10  KM 

0.58 

1 

3 

10  X  20  KM 

0.62 

I 

4 

10  X  30  KM 

0.66 

1 

5 

20  X  30  KM 

0.74 

1 

6 

30  X  30  KM 

0.82 

1 

7 

30  X  40  KM 

0.9 
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V.  Instructional  Feature  Data 


B.  Instructional  Feature 

Cost  and 

Weight 

BENEFIT 

COST 

FEATURE 

1 

WEIGHT 

($K) 

Automated  Performance  Measurema 

nt  1 

1.4 

215.00 

Performance  Indicators 

1 

1.4 

51 . 00 

Procedure  Monitoring 

1 

3.4 

74.00 

Automated  Performance  Alerts 

1 

1.3 

61.00 

Augmented  Feedback 

1 

1.3 

97.00 

Augmented  Cues 

1 

1.3 

155.00 

Record/Playback 

1 

1.42 

92.00 

Total  System  Freeze 

1 

3.1 

24.00 

Remote  Graphics  Replay 

1 

2.7 

76.00 

Initial  Conditions 

1 

4.3 

98.00 

Scenario  Control 

1 

4 . 4 

155.00 

Crash  Override 

1 

4.77 

29.00 

Reset /Repos it ion 

1 

4 . 63 

30.00 

Parameter  Freeze 

1 

2.95 

35.00 

Flight  System  Freeze 

1 

3.8 

35 

Positional  Freeze 

1 

3.3 

35 

Real-Time  Simulation  Variables 

Control  1 

3.7 

137 

Automated  Simulator  Demonstration  | 

2.16 

54 

Adjunct  CAI 

1 

1.7 

236 

Automated  Adaptive  Training 

1 

1.59 

253 

Automated  Cueing  and  Coaching 

1 

1  .  1 

156 
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VI .  Training  System  Data 

A.  Course  and  System  Information 

Variable  |  Value 

- 1 - 

REQUIRED  GRADS/YR  |  800 
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VI.  Training  System  Data 

B.  Model  Information 


VARIABLE 

STANDARD  ADJUSTMENT  RATE 
LEARNING  CURVE  EXPONENT 
MAXIMUM  INST  FEATURE  EFFECT 
MAX  NUMBER  OF  INST  FEATURES 
SETUP  SAVINGS  515 
COST  SAVINGS  WEIGHT 
RECOMMENDATION  BOUNDARY 
ANNUAL  FIXED  COST  FACTOR 
HRLY  VARIABLE  COST  FACTOR 
NEW  DEVICE  UTILIZATION  (HR) 
NEW  DEVICE  LIFECYCLE  (YR) 


VALUE 

30% 

0.7 

10515 

3 

60% 

0.5 

0.5 

300 

80,000 

3000 

10 
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APPENDIX  B 

FOUNDATIONS  AND  DERIVATIONS  OF  OSBATS  MODEL  CONSTRUCTS 


Appendix  B 

FOUNDATIONS  AND  DERIVATIONS  OF  OSBATS  MODEL  CONSTRUCTS 


Introduction 


The  essential  OSBATS  goal  is  to  specify  training  systems 
that  train  students  to  required  performance  levels  at  minimum 
cost.  To  tackle  this  problem  it  is  necessary  to  develop  suitable 
(1)  training-device  learning  curves  and  transfer  functions,  (2) 
cost  functions,  and  (-3)  optimization  techniques.  This  appendix 
provides  supplementary  details  of  the  OSBATS  model  constructs  in 
these  three  areas . 

For  convenience  of  dissertation,  the  simplified  mathematical 
notation  used  during  model  development  is  employed  in  this 
appendix.  This  is,  for  the  most  ?art,  different  from  that  in  the 
body  of  the  report.  Also,  some  variables  are  defined  differently 
depending  on  context.  A  glossary,  provided  at  the  end  of  this 
appendix,  defines  variables  and  parameters  and  identifies  the 
long  form  notation  used  in  the  main  report. 

References  are  also  found  at  the  end  of  the  appendix. 


Medium  Effectiveness;  Learning  Curve  and  Transfer  Functions 


Summary 

There  are  a  number  of  goals  in  learning  cuirve  and  transfer 
function  selection  and  development.  First,  a  learning  curve  must 
be  able  to  represent  actual  student  performance  increase  with 
reasonable  accuracy,  as  must  a  transfer  function  represent 
performance  transfer  to  actual  equipment.  Second,  it  must  be 
possible  to  develop  procedures  to  relate  learning  curve  and 
transfer  function  parameters  to  training  system  design  decision 
variables.  Third,  learning  curves  and  transfer  functions  must  be 
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efficiently  calculable,  since  they  will  be  evaluated  a  great  many 
times  in  the  various  OSBATS  modules.  Fourth,  learning  curves 
should  have  closed  form  inverse  functions,  since  these  are  used 
in  other  methodology  development;  these  learning  curve  inverse 
functions  should  alsj  be  efficiently  calculable. 

One  simplifying  assumption  is  made  at  this  point,  namely, 
that  the  per-^ormance  scales  for  the  various  devices  and  actual 
equipment  t.re  defined  in  such  a  way  that  transfer  functions  from 
training  devices  to  actual  equipment  are  simple  linear,  i.e-, 
that  the  performance  score  on  actual  equipment  will  be  the 
performance  score  on  the  training  device  multiplied  by  a  transfer 
ratio . 

A  methodology  is  then  required  for  determining  mediiim 
(training  device)  effectiveness,  i.e.,  for  (1)  constructing  the 
learning  curve  Pci(t)  representing  the  learning  on  the  device  as 
measured  on  a  normalized  (0-1)  scale,  and  (2)  determining  the 
transfer  ratio  a^.  from  that  device  to  the  actual  equipment,  so 
that  a  learning  curve  P(t)  =  a^  Pd(^)  will  represent  the  learning 
on  the  training  device  in  terms  of  the  performance  a  student 
would  achieve  on  the  actual  equipment  after  training  only  on  the 
training  device.  What  seems  particularly  appealing  is  the 
decomposition  of  the  process  in  such  a  way  that  P(t)  is  a 
function  P( t;s,m,aj-,c,k)  of  parameters  s,  m,  a^,  c,  and  k,  which 
may  be  determined  sequentially  (inverse  order)  as  follow: 

(1)  k  is  a  learning  curve  shape  parameter,  viz.,  the 
exponent  in  a  power  function  form,  and  will  be  the  same 
as  that  for  the  actual  equipment;  k  may  either  be 
based  on  the  literature  (e.g.  Card,  Moran,  and  Newell, 
1983)  or  on  a  fit  to  empirical  learning  data  for  actual 
equipment . 

(2)  c  is  another  learning  curve  shape  parameter,  and 
serves,  further,  as  a  scaling  constant  used  to  convert 
time  units  to  dimensionless  units;  c  will  be  the  same 


as  for  actual  equipment  and  will  be  based  on  a  fit  to 
empirical  learning  data  for  actual  equipment. 

(3)  a^/  the  transfer  ratio,  will  be  a  function  of  the 
medium  fidelity  levels  {r^}  in  the  different  fidelity 
dimensions,  indexed  by  i,  together  with  SME  estimates 
of  the  target  fidelity  levels  {Ri)  for  a  training 
device  to  train  the  task.  If  all  fidelity  levels  are 
met  exactly  then  the  task  can  be  trained  to  a  level 
Y+x(l-Y)  exceeding  the  standard  Y  by  a  specified 
amount,  where  the  fraction  x  is  a  constant.  Prior  to 
the  determination  of  a^,  a  target  transfer  ratio  aR  is 
determined  according  to  the  {Ri>  definition  chosen.) 

The  algorithm  for  aR  also  may  involve  constants  {fi}, 
giving  for  each  fidelity  dimension  the  fractional 
degradation  in  transfer  that  would  occur  for  any  task 
for  which  that  fidelity  is  applicable,  as  that  fidelity 
level  is  decreased  from  1  to  0. 

(4)  m  is  a  time-multiplier  used  to  account  for  increased 
time  efficiency  engendered  by  instructional  features 
and  reduced  setup  time. 

(5)  s  is  an  effective  head  start,  chosen  so  that  the  entiry 
performance  P(0)  equals  a  specified  value. 

The  determination  process  is  represented  in  Figure  B-1 . 
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Target  Fidelity 


Figure  0-1.  Medium  effectiveness  (learning  curve  and  transfer  ratio)  determination 
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Scale  Invariance 


Dimensional  analysis  considerations  require  the  use  of  a 
scaling  constant  for  time  variables.  This  is  basically  the 
parameter  c  discussed  aoove,  though  it  may  also  include  a  factor 
to  control  the  shape  of  the  curve.  Failing  to  introduce  such  a 
scaling  factor  would  force  the  incorporation  of  messy,  iterative, 
and  basically  unappealing  procedures  when  tackling  the  medium 
effectiveness  estimation  problem.  Without  c  the  learning  curve 
cannot  be  adjusted  for  a  change  in  time  scale.  If  there  is  a 
coefficient  c  preceding  (s+t)  in  the  learning  curve,  then  whether 
we  measure  time  in  minutes  or  hours  makes  no  difference  — 
provided  c  is  adjusted  accordingly.  The  curve  will  be  the  same 
as  before  in  the  sense  that  it  will  take  the  same  length  of  time 
t2-ti  to  go  from  performance  level  y^  to  y2  no  matter  how  time  is 
measured.  This  scale  invariance  also  allows  the  simplified 
incorporation  of  instructional  features  and  time  usage  efficiency 
in  the  medium  learning  curves . 

Formally,  a  learning  curve  is  scale  invariant  if  it  has  the 
form  f(-{c[s+t]},  and  other  parameters).  I.e.,  c,  s,  and  t  always 
appear  together  in  the  equation  in  this  form  c[s+t].  The  other 
parameters  may  include  such  things  as  the  shape  constant  k.  Each 
of  the  following  is  scale  invariant,  by  this  definition: 

Negative  Exponential: 


P(t) 

Hyperbolic: 

P(t) 


1  . 

c(s+t ) 
c(s+t)  +  1 


In  the  same  way  the  learning  curves  associated  with  virtually  any 
of  the  standard  learning  theories  (replacement,  acciimulation, 
strength  (Hull),  all-or-none)  can  be  written  in  a  scale  invariant 
form,  as  can,  too,  most  empirical  fits  (e.g.,  logistic). 
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Power  Functions 


Despite  the  popularity  of  the  learning  curve  forms 
identified  in  the  last  paragraph,  Newell  and  Rosenbloom  (1981) 
have  shown  that  a  multitude  of  learning  achievement  measures 
follow  a  power  function  curve.  Their  strong  empirical  evidence 
and  supporting  rationale  motivated  our  selection  of  a  power 
function  form  for  the  OSBATS  learning  curves  (Sticha,  et.al., 
(1986)).  Actually,  Newell  and  Rosenbloom  used  time-to-perform  on 
a  trial,  errors  to  perform  on  a  trial,  and  linear  combinations  of 
these  as  the  measures  that,  when  plotted  on  a  log-log  plot 
against  time  proved  to  be  so  nicely  linear,  e.g.,  fitting  the 
form: 


log(T)  =  -k  ln(s+t)  +  ln(B) 

where  T  represents  the  performance  measure,  and  where  k  and  B  are 
positive  constants.  But  then 

T  =  B  (s+t)"^  (1) 

from  whence  the  term  power  function  rule^.  Note  that  this  may  be 
written 


T 


tc(s+t) ] 


(2) 


if  c  is  defined  to  equal  Thus  T  may  be  considered  a  scale 

invariant  function  of  t. 


^Newell  and  Rosenbloom  use  trials  n,  not  time  t,  as  their 
independent  variable.  This  is  a  minor  complication  since  there 
is  a  fairly  straightforward  method,  for  adjusting  k  when 
converting  from  one  form  to  the  other. 
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The  Newell  and  Rosenbloom  measure  ranges  between  infinity 
(limit  of  bad  performance)  and  zero  (limit  of  good  performance) 
as  (s+t)  ranges  between  zero  and  infinity.  Our  OPT  performance 
measures,  on  the  other  hand,  are  on  a  scale  of  0  to  1,  from  bad 
to  good.  So  a  conversion  of  some  sort  is  needed.  An  appealingly 
simple  conversion  would  be 


l-P(t)  1 

T(t)  =  -  P(t)  =  -  ,  (3) 

P(t)  1  +  T(t) 


i.e.,  to  have  P  in  the  form 


1 

P(t)  =  - .  (4) 

1  +  [c(s+t)l 


However,  this  is  of  quite  a  different  form  than  the  learning 
curve  used  in  early  OSBATS  work.  That  foimi,  with  a  scaling 
constant  now  added,  is 

P(t)  =  1  -  [1  +  c(s+t)3"^  .  (5) 

It  turns  out  that  if  either  (4)  or  (5)  are  fitted  to  the  entry 
performance  P©  at  t=0  and  criterion  performance  Y  at  t=tg  that 
the  difference  between  the  two  curves  is  only  a  few  percent. 

Thus  there  is  no  practical  reason  not  to  use  form  (5),  and  since 
it  is  closer  to  the  form  originally  embedded  in  developed  OSBATS 
modules,  it  was  chosen. 

The  Learning  Curves  in  OSBATS  Context 

Assuming  that  the  task  in  question  can  be  trained  to  some 
level  on  the  medium  in  question,  we  postulate  a  learning  curve  of 
the  following  form; 

P(t)  =  arPd(t)  (6) 
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where 


P(t)  =  the  (entry)  performance  measured  on  the  actual 

equipment  after  training  an  amount  of  time  t  on  the 
training  device  in  question.  This  performance 
approaches  a^  asymptotically. 

ar  ®  transfer  index  defining  the  fractional  amount  of 
performance,  as  measured  in  the  training  device 
environment,  that  transfers  to  performance  on  the 
actual  equipment.  Assumed  a  simple  proportion. 

Pd(t)  =  the  (exit)  performance,  measured  on  the  training 
device,  after  training  an  amount  of  time  t  on  the 
training  device.  By  definition,  this  performance 
measure  also  approaches  1  asymptotically. 


Consider  a  performance  score  y  in  the  training  environment 
and  in  the  operational  environment.  We  would  like  this  to  have  a 
comparable  meaning  in  each  environment.  If,  for  instance,  that 
performance  of  a  tank  gunner  is  defined  as  the  fraction  of  time 
he  can  hit  a  stationary  target  at  1000  meter  range  within  5 
seconds  of  it  coming  into  view,  then  it  is  convenient  to  use  the 
same  definition  for  a  training  device  employing  computer 
generated  imagery.  Thus  a  performance  of  0.80  has  a  certain 
comparable  meaning  in  each  environment.  It  does  not  mean  that  a 
student  who  has  trained  to  0.80  performance  on  the  training 
device  can  go  out  and  score  0.80  in  the  actual  equipment. 

Rather,  he  will  be  able  to  score  aj-{0.80)  on  the  actual 
equipment . 

Once  this  comparability  is  established,  we  may  address 
instructional  features  and  setup  time  reduction  in  terms  of  time 
shrinkage.  I.e.,  these  factors  may  allow  an  increment  of 
performance  improvement  to  be  achieved  more  quickly  since  every 
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hour  of  training  can  be  used  more  efficiently  and  with  less 
wastage.  More  formally: 


(Time  in  training  device)  =  ^  (Time  in  actual  equipment)  (7) 

i.e.,  for  every  hour  of  training  required  on  the  actual  equipment 
to  progress  from  performance  level  y^  to  performance  level  y2  it 
requires  only  a  fraction  1/m  of  that  time  on  the  training  device. 
E.g.  if  it  takes  an  hour  in  actual  equipment  training  to  go  from 
a  perfomance  score  of. 75  to  .80,  it  will  take,  say,  half  an  hour 
to  do  the  same  on  the  training  device  if  instructional  features 
and  reduced  setup  time  lead  to  a  reduction  ratio  of  1/2,  i.e.,  if 
m=2 . 


If  we  assume  that  the  learning  curve  Pd(t)  the  training 
device  has  the  same  basic  form  as  the  learning  curve  Pa('*^) 
actual  equipment,  and  that  that  form  is  scale  invariant  by  our 
earlier  definition,  then  relation  (7)  above  may  be  cast  as  a 
relationship  that  will  allow  us  to  establish  a  simple 
correspondence  between  th®  scaling  coefficients  in  Pd(t)  and 
Pa(^)*  if  constr^’Q  -he  times  mentioned  in  relation  (7)  as 

the  times  required  to  go  from  performance  level  yi  to  y2,  in  the 
respective  environments,  and  if  we  label  these  times  t^  and  t2 
for  the  actual  equipment  and  tj'  and  t2'  for  the  training  device, 
then  we  may  rewrite  ( 7 )  as ; 


and  then  rewrite  this  as 


(8) 


(8'  ) 


where  the  -1  denotes  the  inverse  of  the  functions  indicated.  But 
since  we  have  assumed  PA(t)  and  Pd(t)  have  the  same  scale 
invariant  form,  we  may  write: 
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P^(t)  =  £({c^[s^+  t]},  and  other  parameters)  (9) 

and 

P^(t)  =  f(-{c^[s^+  t]},  and  other  parameters)  (10) 

where  the  other  pareimeters,  including  such  things  as  the  shape 
constant  k,  are  common  to  both  PA(t)  and  Pci(t)  and  will  be 
suppressed  in  the  remaining  equations.  Then 

’=2  -  -  S;  ‘'^^2)  -  =A  (“> 

and  the  other  times  in  (8)  and  (8')  are  found  similarly. 
Substituting  these  into  (8)  yields 

(12) 

but  this  simplifies  to 


which  is  the  promised  correspondence  between  scaling  constants. 

The  actual  application  of  these  results  is  as  suggested  in 
the  summary.  For  notational  simplicity  we  call  c^  simply  c. 
Then  rather  than  write  C(j  in  the  equation  for  Pd(t),  we  write 


1. 

m 


+  t]})  (14) 

where  s^  is  selected  so  that  P(0)  =  Pg,  i.e.,  Pg  =  arPd(O).  But 
substitution  into  (14)  yields  Pd(0)  =  f({mc[sd  +  0]})  «  Pe/^r/ 
from  which  we  may  conclude  that 


“  f"^(P^/a^) 
me  '  e  r ' 


(15) 
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Remark:  The  requirement  Pd(0)  =  ^e/^ir  warrants  some 
discussion.  This  is  a  direct  result  of  the  assvimption  that 
transfer  of  training  from  the  training  device  to  actual  equipment 
is  a  linear  function  of  perfoimiance  on  the  training  device. 

Since  the  training  device  will  certainly  have  less  than  perfect 
transfer  (i.e.,  a^  <  1)/  it  follows  that  Pd(0)  >  Pe»  the 

entering  student  would  be  able  to  score  better  on  the  training 
device  than  on  the  actual  equipment.  This  is  not  a  particularly 
unreasonable  assumption/  but  neither  is  it  obviously  true, 
especially  when  quantified  strictly  with  the  l/a^  ratio.  There 
may  be  times  when  this  conflicts  with  the  desired  "natural" 
comparability  interpretation  between  PA(t)  and  Pd{t)  discussed 
earlier.  In  that  case  we  must  either  abandon  the  linear  transfer 
assumption  or  the  strict  comparability  interpretation.  The 
latter  compromise  is  preferred,  since  it  only  involves  rescaling 
the  scoring  instrument  for  one  or  the  other  environments,  rather 
than  a  major  refoimiulation  of  the  OPT  models.  A  continuation  of 
the  tank  gunner  trainer  example  may  prove  useful.  If  the  typical 
student  could  score  0.2  in  a  real  tank  with  virtually  no  new 
training  (e.g.,  just  a  few  minutes  spent  with  a  manual),  and  if 
the  training  device  is  deemed  to  provide  0.8  transfer,  then  the 
entering  student  should  be  able  to  score  Pd(0)  =  0.2/0. 8  =  0.25 
on  the  training  device  with  virtually  no  training  (e.g.,  with 
just  a  few  minutes  with  a  manual).  If  this  is  not  the  case,  it 
needs  to  be  made  the  case,  i.e.,  the  scoring  instriiment  is 
revised  so  that  the  average  entering  student  does  score  0.25. 

We  propose  that  the  time-multiplier  m  be  partially  based  on 
considerations  of  the  effective  use  of  time  in  the  actual 
equipment  versus  the  training  device  environments,  and  partially 
on  the  benefits  of  instructional  features.  Suppose  training  a 
student  on  take-off  and  landing  tasks  entail  only  20  useful 
minutes  out  of  every  hour  training  session,  the  remaining  time 
being  spent  on  transportation  to  the  airfield,  pre-flight 
checkout,  and  holding  pattern  cycling — activities  that  constitute 
overtraining  of  mastered  tasks,  or  just  wasted  time.  Let  u^  = 
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20/60  represent  the  ratio  of  useful  to  total  time  on  actual 
equipment.  Suppose,  further,  that  50  minutes  of  every  hour  are 
deemed  useful  on  the  training  device  in  question.  Let  U(i=50/60 
represent  the  the  ratio  of  useful  to  total  time  on  the  training 
device.  Then  it  seems  the  training  device  has  an  m  =  u^/ua  = 
50/20  advantage  over  the  actual  equipment,  in  terms  of  the 
usefulness  made  of  time. 

We  argue  that  instructional  features  provide  a  similar  time- 
multiplier  effect  by  enhancing  every  hour  in  the  training  device 
in  terms  of  concepts  presented,  effective  trials  executed,  and 
reinforcement  and  feedback  provided.  Let  be  an  index  of  any 
such  instructional  features  benefit  in  the  training  device. 

(E.g.,  Id=1.0  represents  no  enhancement,  and  1^=1. 5  represents  a 
50%  better  use  of  time;  we  expect  modest  enhancements  for  most 
training  devices,  e.g.,  1^  =  1.2.)  Then  in  addition  to  the  time- 
multiplier  effects  occasioned  by  reduced  time  wastage,  the 
instructional  features  are  included  in  m; 

Ud 

m  =  Id  —  .  (20) 

As  an  illustration  of  the  interpretation  of  m,  consider  the 
training  of  takeoff  and  landing  on  a  recreational  flight 
simulator  for  a  personal  computer  versus  that  in  an  actual  Piper 
Cub  aircraft.  To  learn  this  on  an  actual  Piper  Cub^  we  would 
need  to  go  to  the  airport,  hire  an  instructor,  wait  for  the 
aircraft  to  be  readied,  be  given  a  briefing  on  aircraft 
instrumentation,  told  the  basics  of  flight,  wait  for  runway 
clearance,  circle  around  between  takeoff  and  landing,  etc. 


^We  are  simply  envisioning  the  rudimentary  acquisition  of 
the  skill  —  not  a  certification  program  requiring  the  mastery  of 
navigation,  regulation,  flight  physics,  as  well  as  the  actual 
development  of  the  take-off  and  landing  skills.  At  the  level  of 
skill  we  are  envisioning  we  would  be  uncertified  and  unwise  to 
attempt  takeoff  or  landing  without  an  instructor  at  dual 
controls . 
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Takeoff  and  landing  is  not  too  hard  on  an  actual  Piper  Cub  (we 
are  told)  so  suppose  that  in  a  6  hour  day  at  the  airport  we  are 
able  to  acquire  the  rudimentary  skill,  e.g.,  to  go  from  a 
completely  green  performance  score  of,  say,  0.05,  to  score  of  0.5 
on  an  appropriate  scoring  instrument.  Suppose  that  during  that  6 
hours  we  spend  2  hours  actually  being  instructed  on  takeoff  and 
landing,  observing  the  instructor  perform  takeoff  and  landings, 
attempting  the  task  ourself  at  our  dual  controls,  and  receiving 
feedback  and  critique.  Then  u^  »  2/6.  In  contrast,  in  learning 
to  take  off  and  land  in  a  simulated  Piper  Cub  on  our  PC-based 
recreational  flight  simulator  aJjnost  all  our  time  is  devoted 
directly  to  learning  the  task  by  reading  the  manual  or  by 
actually  attempting  take  offs  and  landings.  Thus  a  useful  time 
ratio  ud=50/60  might  be  reasonable  for  this  environment.  In 
addition,  the  simulated  environment  offers  some  significant 
instructional  feature  advantages  over  the  actual  environment.  We 
can  intentionally  make  mistakes  in  order  to  get  a  feel  for 
aircraft  response,  stall  speed,  rates  of  descent,  and  so  forth. 
Crashing  the  plane  offers  reinforcement  not  desirable  with  actual 
equipment.  The  value  of  such  instructional  features  is  difficult 
to  estimate,  but  let  us  say  1^=1. 20.  Then  the  time-multiplier 
for  the  PC-based  simulator  is  m  *  Id(^d/^A)  ®  1 . 2[ ( 50/60 )/( 2/6 ) ] 
=  3.0.  Since  it  takes  6  hours  to  acquire  rudimentary  take  off 
and  landing  skill  on  actual  equipment,  it  will  only  take  6/3  =  2 
hours  to  acquire  the  rudimentary  skill  on  the  simulator  (i.e.,  to 
go  from  entry  level  performance  Pg/a^  to  a  score  0.5  on  a 
comparable  scoring  instrument)  under  our  assumptions.  We  should 
expect  the  transfer  ratio  from  our  PC-based  flight  simulator  to 
the  actual  aircraft  to  be  very  poor,  since  the  fidelity  is 
extremely  poor  in  every  dimension;  perhaps  aj.=0.20  would  be  a 
reasonable  estimate.  (Estimation  of  transfer  of  training  is  the 
subject  of  the  next  section.)  So  we  would  only  be  able  to  score 
(0.2)(0.5)  =  0.1  on  an  actual  aircraft  after  spending  2  hours  on 
the  simulator.  Further,  we  could  never  attain  the  0.5  score 
achieved  in  6  hours  on  actual  aircraft  by  training  instead  only 
on  the  simulator.  A  perfect  1.0  rating  on  the  simulator  only 
translates  to  a  0.2  rating  on  the  actual  aircraft. 
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Transfer  of  Training 

Our  approach  to  determining  the  transfer  of  training  for  a 
medium  involves  two  steps.  First,  we  determine  the  transfer 
index  a^  for  the  task  in  question  for  a  mediiun  that  serves  as  a 
benchmark  in  the  sense  that  it  meets  all  the  fidelity 
requirements  exactly.  Second,  we  degrade  that  transfer  index  to 
attain  the  actual  transfer  for  the  mediiim  under  consideration, 
which  may  be  in  some  ways  better  and  in  other  ways  worse  than  the 
benchmark  medium.  Both  of  these  steps  are  based  on  measures  of 
fidelity  relative  to  the  actual  equipment  and  operational 
environment.  The  fidelity  dimensions  are  indexed  by  i.  The 
benchmark  medium  has  fidelities  labeled  Ri,  ...,  Rn-  The  medium 
in  question  has  fidelities  labeled  r^,  ...,  rn. 

Benchmark  Fidelities.  An  essential  part  of  the  OSBATS  data 
base  is  the  definition  of  the  benchmark  medium  in  terms  of 
fidelity  levels.  In  work  to  date  we  have  called  these  the 
fidelity  requirements  for  the  task.  Originally  we  defined 
fidelity  requirements  as  follows: 

Definition  It  Required  fidelity  represents  the  minimum 
level  of  realism  fin  the  various  fidelity  dimensions^ 
required  to  train  the  task  to  standard  on  the  training 
device. 

Mathematically,  this  was  interpreted  to  mean  that  the  learning 
curve  P(t)  had  asymptote  at  the  standard  Y: 

lim  P(t)  *  y  .  (21) 

t->Inf inity 


But,  as  is  clear  from  (6),  P(t)  also  approaches  a^,  the  transfer 
index  from  the  training  device  to  the  actual  equipment,  so  this 
definition  was  equivalent  to  saying  that  the  transfer  index  is 
equal  to  the  training  standard  if  all  fidelity  requirements  are 
just  met: 
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aR  =  Y. 


(22) 


This  definition  was  fairly  simple  and  was  dependent  only  on 
the  nature  of  the  task,  not  on  any  assumed  features  of  the 
training  device  (instructional  features,  reduced  setup  time, 
etc.).  However,  it  seemed  to  us  that  the  statement  "fidelity 
requirements  met"  should  imply  that  the  device  should  be  able  to 
train  to  the  standard  in  a  finite  amount  of  time,  rather  than 
simply  approaching  it  asymptotically.  Therefore  a  second 
definition  was  considered; 

Cefinition  2.  Required  fidelity  represents  the  level 
of  realism  needed  to  train  the  task  to  standard  on  the 
training  device  in  the  same  time  as  on  actual 
equipment,  assuming  that  the  training  device  eliminates 
most  (nominally  60%^  of  the  setup  time  and  will  have 
instructional  features  causing  further  reduction 
(nominally  10% 1  in  the  time  required  to  train  tasks  on 
the  device  as  compared  to  on  actual  equipment.  Thus. 
i'^  fidelity  requirements  are  met,  the  increased  time  to 
train  to  standard  due  to  lower  fidelity  is  just  offset 
by  instructional  features  and  device  time  savings  due 
to  reduced  setup  time. 

Mathematically  this  was  equivalent  to  the  requirement  that 

P(ts)  =  Y  (23) 

where  ts  is  the  time  required  (including  setup)  to  train  the  task 
on  actual  equipment.  This  definition  had  the  advantage  of 
allowing  standard  achievement  in  a  finite  amount  of  time  on  a 
training  device  meeting  the  fidelity  requirements.  However,  it 
had  five  major  disadvantages:  (1)  it  was  difficult  to  understand 
and  to  communicate  to  a  SME;  (2)  it  required  consideration  and 
knowledge  of  setup  time  on  the  actual  equipment;  ( ;; )  it  required 
an  assumption  on  the  nature  of  the  training  devi  in  terms  of 
setup  time  savings  and  instructional  features  benetxts;  (4)  it 
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became  more  difficult  and  time  consvuning  to  solve  for  the 
learning  curve  parameters;  (5)  on  a  task  for  which  negligible 
setup  time  is  required  on  actual  equipment,  and  for  a  device 
without  instructional  features,  essentially  perfect  fidelity 
would  be  required,  which  seems  unreasonable.  Therefore,  this 
definition  was  ruled  out. 

So  we  reconsidered  definition  1.  Perhaps  it  would  be 
workable  if  we  were  just  careful  with  the  associated  words 
defining  the  meanings  of  the  fidelity  "requirements".  Since  a 
training  device  just  meeting  the  fidelity  "requirements"  takes 
forever  to  reach  the  training  standard,  the  medium  selection 
methodology  will  end  up  selecting  last  stage  media  (prior  to 
actual  equipment)  with  some  fidelity  requirements  exceeding  these 
"requirements".  Is  this  really  objectionable?  It  just  defines  a 
floor.  Media  with  lesser  fidelity  might  still  be  selected  for 
use  in  earlier  training  stages,  or  even  for  the  final  stage  if 
other  fidelity  dimensions  compensated.  So  it  was  suggested  that 
we  go  back  to  definition  1,  but  talk  about  fidelity  "floor",  or 
"base",  or  some  other  more  accurate  term. 

But  the  counter  argument  held  the  day:  that  a  SME  would  be 
uncomfortable  estimating  such  "floor"  requirements — that  he  or 
she  would  be  better  able  to  estimate  fidelities  actually  needed 
to  train  the  task  to  criterion  on  the  training  device.  So  we 
proposed  a  variation  on  definition  1,  namely  by  choosing  a 
transfer  index  equal  to  some  level  of  performance  higher  than  the 
standard,  and  defining  fidelity  requirements  so  that  the  transfer 
equals  that  level  of  performance.  In  particular,  we  proposed: 

Definition  3i  Required  fidelity  is  achieved  by  a 
fidelity  vector  (R^,  ....  such  that  the  associated 

transfer  index  ap  exceeds  the  training  standard  Y  by  a 
fraction  x  of  the  distance  to  the  standard,  i.e., 

aR  =  Y  +  x(l  -  Y).  (24) 
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This  definition  allows  the  actual  standard  to  be  achieved  in  a 
finite  amount  of  time  on  a  device  just  meeting  the  requirements. 
It  introduces  another  parameter,  x,  but  this  extra  pareuneter  is 
advantageous  since  it  allows  us  to  adjust  to  the  SME 
interpretation  of  the  {Ri)  rather  than  requiring  us  to  come  up 
with  wording  and  instructions  to  force  the  SME  to  our 
interpretation  of  the  {Ri}*  We  just  try  different  values  of  x  to 
come  up  with  something  that  corresponds  to  SME  interpretation  of 
the  term  "fidelity  requirements"  (or  whatever  other  term  we 
choose  to  indicate  the  {Ri}).  Should  x  turn  out  to  be  0,  for  the 
terminology  used,  then  Definition  3  corresponds  exactly  to 
Definition  1. 

Other  approaches  to  defining  the  {Ri)  can  be  suggested^,  but 
for  now  we  will  go  with  Definition  3.  The  advantages  are,  to 
recapitulate:  (1)  allows  the  SME  to  interpret  our  "fidelity 

requirement"  terminology  as  he  or  she  chooses  (we  will  adjust  x 
appropriately);  (2)  depends  on  the  task  to  be  trained  only — no 
assumptions  are  made  regarding  features  of  the  actual  training 
devices;  (3)  decouples  training  transfer  from  learning  curve 
parameters,  consistent  with  the  plan  outlined  at  the  beginning  of 
this  paper,  thereby  eliminating  the  complex  and  time  consuming 
iteration  occasioned  in  some  of  our  earlier  formulations. 

Transfer  Index  Degradation.  Let  R^  be  the  "required"  (or 
"benchmark")  fidelity  (i-th  dimension)  for  a  training  device 
under  consideration  to  train  the  task  in  question,  according  to 
SME  judgment.  Let  r^  be  the  actual  fidelity  (i-th  dimension)  for 
the  training  device  in  question.  Assume,  for  now,  that  we  are 
looking  for  a  function 

^One  approach  suggested  defining  the  (Ri)  such  that  if  the 
corresponding  fidelities  in  the  actual  equipment  were  degraded  to 
those  levels  (down  from  1.0)  then  the  task  training  would  take 
twice  (or  some  other  set  multiple)  as  long. 

Yet  another  approach  suggested  assuming  that  the  SME  were 
generous  in  estimating  each  "required  fidelity",  so  we  would 
degrade  the  SME  estimate  by  some  fixed  percentage  to  obtain  the 
(Rj^}  to  be  used  with  Definition  1. 
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following  conditions; 

( 1 )  aj.  =  0  if  any  ri  =  0  whan  th©  corrasponding  >  0 , 
i.©.,  if  a  fidelity  dimension  is  included  at  all,  then 
it  is  assumed  that  some  fidelity  in  this  dimension  is 
absolutely  essential.  (This  condition  is  arguable,  and 
a  variation  on  it  will  be  considered  later.) 

(2)  a^  =  aR  if  each  ri  =  Ri.  (I.e,,  any  device  with  the 
same  fidelity  measures  is  indistinguishable  from  the 
benchmark  device  defined  by  the  SME,  insofar  as 
transfer  of  training  is  concerned.) 

(3)  ar  =  1  if  each  ri  =  1,  i.e.  perfect  fidelity  is 
achieved.  (In  this  case  the  device  is  indistinguish¬ 
able  from  the  actual  equipment,  insofar  as  transfer  of 
training  is  concerned.) 

Simple  Multiplicative  Axis-Anchored  Degradation  Function.  A 
very  simple  function  that  satisfies  the  requirements  is^ 

r'-r—r  “iQ  ^R 

©r  =  [_J^  ^ij  '  where  Q  =  -  .  (25) 

i 

where  the  "pi"  indicates  a  mathematical  product  of  the  terms 
following . 

Relaxed  Axis-Anchored  Multiplicative  Degradation  Function.  A 
related  functional  form  uses  the  required  (benchmark)  fidelity 
dimensions  as  exponents  to  give  a  certain  "softening"  in  response 
to  degradation  in  less  demanding  requirements.  The  functional 
form  is 


In  a 


where  Q  = 


R 


rTRi^'i 

i 


(26) 
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The  softening  is  shown  by  differentiating  both  (25)  and  (26) 
with  respect  to  ri*  In  the  first  case, 

dr^  °  ' 

while  in  the  second  case, 

da  a 

—  =  Q  R.  —  . 

dri  r. 

These  equations  are  more  useful  for  this  discussion  when 
rewritten  in  terms  of  the  fractional  change  (dr^/r^)  in  r^,  and 
the  associated  fractional  change  (da^/a^-)  in  a^,  i.e.,  the  ratios 
of  the  differential  elements  to  their  respective  variables.  For 
the  simple  multiplicative  axis-anchored  degradation  function,  the 
relationship  is 


(27a) 


while  for  the  relaxed  axis-anchored  multiplicative  degradation 
function  the  relationship  is 


da  dr^ 

—  «  Q  Rd  — 
^r  ^i 


(27b) 


Thus,  a  5%  change  in  r^  will  effect  the  same  percentage  change 
(5%  of  Q)  in  aj.  regardless  of  the  value  of  Ri,  for  the  simple 
multiplicative  axis-anchored  degradation  function,  but  will 
effect  a  percentage  change  (5%  of  Q  Ri)  in  a^  proportional  to  Ri 
for  the  relaxed  axis-anchored  multiplicative  degradation 
function.  Thus,  a^  will  be  less  sensitive  to  percentage 
shortfalls  in  Ri  when  Ri  is  low,  in  the  latter  case. 


Un-Anchoring  at  the  Axes .  Suppose  that  condition  (1)  is 
replaced  by  conditions  on  each  fidelity  dimension  indicating 
whether,  and  to  what  degree,  degradation  is  limited  as  that 
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fidelity  is  driven  to  zero.  Let  us  formalize  this  by  the 
following  rule:  Hold  rj  =  Rj  for  each  fidelity  dimension  j 
except  one,  i.  Consider  the  response  in  to  variations  in  r±. 
As  ri  varies  from  1.0,  down  through  R^,  and  on  down  to  0,  the 
value  of  a^  varies  from  some  maximum  value,  ari"*",  to  a^,  to  a 
minimum  value,  ari".  Require  that  ar£"/ai.i‘*'  equal  a  fraction  f^, 
characteristic  of  that  fidelity  dimension  and  the  task,  or 
general  type  of  task  to  be  trained.  Setting  each  f^  =  0  is 
equivalent  to  restoring  condition  (1). 

The  new  rule  allows  certain  fidelity  dimensions  to  be  deemed 
less  critical  than  others.  For  instance,  it  seems  that  for  most 
tasks  audio  fidelity  is  more  of  an  enhancement  than  an  essential. 
It  might  therefore  be  reasonable  to  set  f^  =  0.3,  say.  Although 
in  principle  there  will  be  a  distinct  set  of  {fi}  for  each  task, 
it  is  hoped  that  the  f^  will  be  not  vary  over  large  groups  of 
tasks,  so  that  SHE  estimation  workload  will  not  be  substantially 
increased.  This  hope  is  bolstered  by  the  following 
interpretation  of  the  {fi),  which  can  be  shown  to  be  tirue:  f^  = 
a^  if  r^  *  0,  and  rj  =  1  for  j  i.  But  since  a  device  with  rj 
=  1  for  all  j  is  indistinguishable  from  actual  equipment  in 
regards  to  training,  this  suggests  a  way  to  more  concretely  think 
about  fi.  Suppose  a  training  device  is  constructed  by  taking 
actual  equipment,  but  somehow  denying  the  student  the  i-th 
fidelity  dimension;  then  fi  is  the  transfer  of  training  that 
would  occur  from  this  device  to  actual  equipment.  (In  order  to 
avoid  safety  implications,  assume  that  an  instructor-operator  has 
dual  controls,  as  needed.)  For  instance,  if  the  student  were 
giver  heavy,  padded  headphones  that  prevented  helicopter  noise 
from  being  heard,  yet  still  permitted  communication  with  an 
instructor,  how  well  would  an  actual  helicopter  train: 

(1)  Instrument  navigation?  (Answer:  Fine,  f^  =  1.) 

(2)  Take-off  and  landing?  (Answer:  Reasonably  well, 
though  some  useful  information  contained  in  engine  and 
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rotor  noise  and  would  be  unavailable  in  this  training. 
So  perhaps  =  .6.) 

(3)  Engine  or  drive  train  malfunction?  (Answer:  Very 
poorly.  This  kind  of  malfunction  is  very  often 
determined  and  diagnosed  by  sound,  so  perhaps  f^  =  0.) 

We  might,  similarly,  consider  how  well  the  student  training  would 
transfer  if  his  training  were  conducted  with  the  windows  covered 
with  gauze,  giving  the  effect  of  highly  impaired  visual 
resolution,  and  thereby  estimating  the  associated  f^.  If  we 
envision  rigging  up  mechanical  assists  or  detriments  to  destroy 
the  proper  control  loading  feel  of  actual  equipment  control 
actuators  (stick,  levers,  knobs),  then  we  may  estimate  fi  for  the 
control  loading  fidelity  dimension.  Shuttering  the  windows 
immediately  after  weapon  firing  and  before  warhead  impact  would 
provide  us  a  basis  for  estimating  f^  for  the  weapon  effect 
fidelity  dimension.  Not  all  fidelity  dimensions  can  be  addressed 
in  this  way?  one  cannot  effectively  degrade  or  deny  motion  cues 
in  actual  equipment,  for  instance. 

/ 

The  particular  degradation  function  associated  with  this  new 
condition  is: 


a 


r 


=  q 


(r^  +  b.) 


R 


■] 


(28) 


where  q,  Q,  and  the  {bi}  are  parameters  selected  so  that  a^ 
satisfies  the  operable  conditions,  viz.. 


(1')  For  each  i,  a^i"  =  fi  ari'*’,  where  a^i”  is  the  transfer 
associated  with  rj,  =  0  and  where  ari"*"  is  the  transfer 
associated  with  ri  =  1,  when  rj  =  Rj  for  each  fidelity 
dimension  j  other  than  i. 


(1")  [Equivalent  to  (1')].  If  ri=  0,  and  rj  =  1  for  each  j 
i,  then  Sj.  =  fj^. 
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(2)  aj.  =  Sr  if  each  =  R^.  (I.e.,  any  device  with  the 

same  fidelity  measures  is  indistinguishable  from  the 
benchmark  device  defined  by  the  SME,  insofar  as 
transfer  of  training  is  concerned.) 

(3)  a^  =  1  if  each  r^  =  1,  i.e.  perfect  fidelity  is 
achieved.  (In  this  case  the  device  is 

indistinguishable  from  the  actual  equipment,  insofar  as 
transfer  of  training  is  concerned.) 


The  parameters  are  found  in  the  order  Q,  {b}^,  q,  as  follow. 
First,  Q  is  found  in  order  to  force 


r”  -i-n 

..-IT 

[r,  t  (1  .  Ri)fi''^  J 

(Unfortunately,  this  requires  an  iterative  root  finding  approach. 
However,  it  is  needed  only  to  find  the  parameters  for  the  a^ 
curve,  not  for  each  evaluation  of  a^. )  Next,  find  each  of  the 
<bi},  using  the  formula; 


*>1 


The  fitting  is  completed  by  finding  q  from 


q  = 


(1  +  b.) 


(30) 


(31) 
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A  short  exercise  in  algebra  would  show  that  if  each  fi 
equals  0  that  this  formulation  collapses  to  that  of  the  relaxed 
axis-anchored  degradation  function.  This  is  quite  reasonable, 
since  in  that  case  condition  (1')  or  its  equivalent  (1") 
degenerate  to  the  axis -anchoring  condition  (1)  presented  earlier. 

Transfer  Degradation  Based  on  Analogy  to  Utility  Theory.  We 
also  investigated  the  use  of  degradation  formulas  from  utility 
theory,  as  developed  by  Keeney  and  Raiffa  (1976),  since  there, 
too,  one  is  estimating  a  function  between  0  and  1  on  some  space. 
We  hoped  that  some  of  the  independence  conditions  that  prove 
useful  in  utility  theory  would  have  meaningful  analogues  in  the 
fidelity  degradation  problem.  However,  we  have  not  been 
successful  in  these  attempts.  It  appears  that  there  are  inherent 
conflicts  in  the  two  formulations.  Further  research  along  these 
lines  is  being  held  in  abeyance. 

Detailed  Computational  Example 

We  conclude  the  discussion  of  measurement  effectiveness 
modeling  with  a  detailed  example  employing  modified  power 
function  learning  curves  and  using  a  degradation  function  that  is 
not  anchored  at  all  to  the  axes.  Suppose  there  are  two  fidelity 
dimensions  of  interest.  Suppose  that  the  following  estimates  are 
provided  by  the  SME  and/or  design  engineers: 

Required  Estimated 


( Benchmark ) 

Minimum 

Training  Device 

Fidelity  Fidelity 

Performance 

Fidelity 

Dimension  Level 

Parameter 

Level 

i  Ri 

fi 

ri 

1  0.90 

0.00 

0.60 

2  0.70 

0.30 

0.40 

Task  Training  Standard: 

Estimated  Entry  Level  Performance: 

Y  =  0.75 

Pe  =  0.10 

Time  to  Achieve  Standard  on  Actual 
Equipment : 

Useful  Time  Ratio,  Actual  Equipment: 

ts  =  10.0  hrs 

Ua  =  0.20 
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Useful  Time  Ratio,  Training  Device:  =  0.80 

Instructional  Features  Factor, 

Actual  Equipment:  =  1.00 

Instructional  Features  Factor, 

Training  Device:  Id  =  1.25 

Further,  suppose  that  through  experience  the  OSBATS  modelers  have 
established  that  the  parameter  value  k  =  1.25  causes  the 
learning  curves  to  have  a  "shape"  that  seems  to  describe  the  task 
training  processes  being  investigated.  Also,  suppose  that  they 
have  established  that  the  parameter  value  x  =  0.3  seems  to  give 
proper  meaning  to  the  SHE  estimates  of  "required"  fidelity. 

Following  the  upper  path  of  the  IDEF  diagram  shown  at  the 
beginning  of  this  memorandum,  we  first  find  from  (24)  that 
aR  =  0.825. 


Next,  the  parameters  Q,  b^,  b2/  and  q  for  the  a^  relation 
are  found.  Solving  (30)  with  a  root  finding  procedure  yields  Q  = 
0.5669.  From  (31),  we  find  that  bi  =  0  and  b2  =  0.0176.  From 
(31)  we  then  find  that  q  =  0.9931.  Then  the  transfer  function 
for  training  devices  as  a  function  of  the  fidelity  levels  ri  and 
r2  is  found  from  (28): 


a^  =  0.9931  [^(r^  +  0)^*^(r,+  0.0176) 


0.5669 


where  the  exponents  0.9  and  0.7  are  Ri  and  R2/  respectively.  A 
graph  of  a^  as  a  function  of  r^  for  various  levels  of  r2  is 
presented  in  Figure  B-2. 

We  have  assiimed  that  the  actual  equipment  has  a  learning 
curve  described  by  a  modified  power  function  (5): 

P^(t)  =  f({c  [s^+  t]}) 

=  1  -  [1  +  c(s^+  t)]’^ 
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where  f  has  the  functional  form 


f(z)  *  1  -  [1  +  z]  ^ 

and  inverse 

z  =  f-l(y)  =  (1  -  -  1  . 


We  have  established  that  k  =  1.25,  so  we  may  determine  the  value 
of  c  that  causes  PA(t)  to  achieve  the  training  standard  Y  =  0.75 
after  time  ts  =  10.0  hours  from  entry.  To  do  this  we  note  that 


c(s.  +  0)  =  (1  -  -  1 


and 


c(Sa  +  tg)  =  (1  -  -  1 


which  we  may  solve  simultaneously  to  yield  the  basic  time 
scaling  factor: 

(1  -  -  (1  - 

c  =  - - - —2 —  =  0.194 

s 


and  the  head  start  for  actual  equipment: 


(1  -  Pe) 


-1/k 


-  1^  =  0.452. 


We  assume  that  the  "local"  performance  curve  P(i(t)  for 
the  training  device  has  the  form 

P^(t)  =  f({mc  [s^+  t]}) 

=  1  -  [1  +  mc(s^+  t)]"^ 

where  f()  is  the  same  as  for  actual  equipment.  We  determine  the 
the  time-multiplier  as: 

u  , 

m  =  I. - -  =  5.00 

a  u. 
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■  r,  Tranafer  Indox  from  Tratnlns  Dovloa 


and  the  head  start  as: 


s  .  =  —  f"^(P  /a  )  =  —  fTl  -  P  /a  -  ll  =  0.183. 

d  me  '  e  me  Ll  e  r' 

Remark:  The  only  iterative  proeedure  needed  in  this  example 
was  the  determination  of  the  parameter  Q  for  the  funetion  a^.  As 
various  training  deviees  options  are  eonsidered,  only  the  {r^}, 
not  the  {Ri},  are  varied,  so  this  iterative  proeedure  need  not  be 
repeated.  This  is  an  improvement  over  some  of  our  previous 
models  in  which  speed  of  computation  was  a  potentially  serious 
problem. 

The  learning  curves  PA(t),  Pd(^)/  Pd{t)  shown 

in  Figure  B-3.  Note  that  Pd(0)  >  PA(0)  =  P(0),  as  discussed 
earlier.  As  expected,  P(t)  provides  the  greatest  training 
benefit  initially,  but  then  becomes  relatively  ineffective  due  to 
the  transfer  index  "ceiling". 


Cost  Functions 


Summary 

Two  types  of  training  device  cost  function  types  are 
employed  in  OSBATS  modeling.  The  Resource  Allocation  module 
employs  piecewise  continuous  cost  functions,  capable  of 
representing  setup  costs,  the  jumps  occasioned  by  discrete 
acquisition  of  major  cost  training  devices,  and  the  implications 
of  training  devices  already  on  hand.  This  type  of  function  can 
also  be  used  to  represent  economies  of  scale,  though  this  has  not 
been  explicitly  addressed  in  work  to  date.  The  detailed  cost 
function  is  based  on  training  system  per  unit  acquisition  costs, 
life  cycle,  fixed  operating  costs,  variable  operating  costs,  and 
hours  that  the  device  may  be  used  annually.  The  relationship  of 
the  cost  function  to  these  constituent  components  will  be 
explained.  The  methodology  for  determining  the  values  of  the 


Porformftnoe  (App: 


Learning  Curves 


m 

C 


m 

C 

u 

0 

o 

n 

• 

a 

1. 

Q. 

0 

k 


0.9 

0.8 

0.7 

0.6 

0.9 

0.4 


0.3 


0.2 


Time,  t  (hrs) 


Figure  B-3.  Learning  curves. 
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constituent  components  themselves  are  not  addressed  in  this 
appendix . 

The  other  modules  employ  a  simple  linear  cost  function 
based  on  a  fi^e  hourly  rate.  This  simple  cost  function  is  an 
approximation  to  the  more  detailed  function. 

Detailed  Cost  Function 

The  piecewise  linear  cost  curve  consists  of  a  series  of 
linear  segments .  In  the  case  that  economies  of  scale  are 
unimportant,  only  the  discrete  acquisition  of  ievices  causes 
deviations  from  simple  linear  curve,  and  results  in  a  cost  curve 
with  an  infinite  series  of  equal  steps.  The  cost  curve  is 
formally  defined  by  the  following  parameters: 

t  -  Expected  total  time  that  an  individual  student  will  use 
the  training  device  in  question  during  his  training 
program. 

C(t)  -  Training  device  costs  attributed  to  each  individual 

student  if  each  student  uses  training  device  for  amount 
of  time  t. 

N  -  Number  of  segments  in  the  cost  curve. 

i  -  Index  of  segment  (interval)  of  cost  curve. 

j  -  Total  number  of  devices  acquired  (for  giv**n  t). 

Ti  -  Starting  value  for  i-th  segment  of  cost  curve  C(t) 

Ai  -  Height  of  cost  curve  C(t)  at  beginning  of  i-th  segment. 

Bi  -  Slope  of  i-th  segment  of  cost  curve  C(t). 
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With  two  exceptions,  each  segment  is  open  on  the  left  and 
closed  on  the  right.  The  first  exception  is  the  first  interval, 
which  is  closed  on  the  left  and  on  the  right.  The  second 
exception  is  the  final  interval,  which  is  open  on  the  left  and 
extends  without  limit  on  the  right.  The  cost  curve  is  defined  as 
follows : 


C(t) 


Ai  +  Bi(t  -  Ti) 
Ai  +  Bi(t  -  Ti) 
An  +  BN(t  -  Tn) 


if  Ti  <=  t  <=  T2 
if  Ti  <  t  <=  Ti+i 
if  Tn  <  t 


(32) 


The  parameters  Ai  and  Bi  may  be  related  to  the  following 
data  or  projections; 


P  -  Number  of  students  trained  annually  in  the  training 
program. 

I  -  Investment  cost  for  each  training  device. 

L  -  Life  cycle  for  each  training  device. 

F  -  Fixed  operations  cost,  annual. 

r  -  Variable  operations  cost,  per  student  hour. 

H  -  Maximum  potential  annual  usage  (student  hours)  of  device 
per  year,  considering  school  operating  hours,  device 
maintenance  schedules,  number  of  students  served 
simultaneously,  etc. 

These  parameters  are  related  to  the  cost  curve  C(t)  by  the 
following  master  equation; 

Total  investment  +  Total  fixed  +  Total  variable 
cost  operating  cost  operating  costs 

C(t)  =  - 

Total  number  of  students  processed 
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where  "total"  nieans  across  all  students  and  all  years  in  the 
device  lifetime. 

The  total  number  of  students  processed  over  the  life  cycle 
of  the  training  device  is  just  PL. 

Suppose  that  each  student  uses  the  kind  of  training  device 
in  question  for  t  hours.  Then  since  P  students  are  processed 
annually,  the  training  device  will  be  used  a  total  of  Pt  hours 
annually.  But  each  particular  training  device  is  available  for 
only  H  hour  a  annually,  so  the  total  number  of  devices  required  is 


j  = 


Pt/H 

integer (Pt/H)+1 


if  Pt=jH  (exact  multiple) 
otherwise 


(34) 


This  will  remain  in  effect  for  an  interval  of  length  H/P,  at 
which  time  j  is  incremented.  So  the  steps  will  occur  at  T^  =  0, 
T2  =  Pt/H,  ...,  Tn  =  (N-l)Pt/H.  ^ 

If  some  training  devices  of  the  type  in  question  have 
already  been  acquired,  then  the  cost  analyst  will  want  to  make 
sure  that  the  steps  associated  with  acquisition  of  these  first 
devices  properly  reflect  both  sunk  investment  cost  and  sunk 
(obligated)  fixed  costs.  Essentially,  this  will  remove  the  steps 
in  the  cost  curve  that  would  otherwise  have  been  associated  with 
the  acquisition  of  the  devices  now  on  hand.  This  modification 
will  be  quite  important  for  proper  operation  of  the  Resource 
Allocation  module,  since  will  result  in  a  proper  tendency  to 
allocate  more  heavily  to  these  devices.  Because  the  present 
OSBATS  data  base  assumes  assets  on  hand  are  zero,  this  refinement 
will  not  be  discussed  further  here. 

If  j  devices  are  acquired,  the  total  investment  cost  is  jl 
(since  the  j  training  devices  are  purchased  only  once  during  the 
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life  cycle).  The  total  fixed  annual  operating  cost  is  jFL(since 
the  annual  fixed  operating  costs  must  be  expended  for  each  device 
and  each  of  the  years  in  the  device  life  cycle).  The  variable 
operating  cost  is  PLrt  (since  the  variable  operating  cost  rate  r 
is  imposed  only  for  the  PLt  student  hours  expended  over  the  life 
cycle  of  the  training  cevice). 

Then  the  total  cost  is 
jl  +  jFL  +  PLrt 

C(t)  =  -  (35) 

PL 

Assume  that  the  first  segment  of  the  cost  curve  is  simply 
the  single  point  at  (0,0)  corresponding  to  no  devices  being 
acquired.  The  second  segment  of  the  cost  curve  then  corresponds 
to  one  device  being  acquired.  In  general,  the  i-th  segment  will 
correspond  to  i-1  devices  being  acquired,  i.e., 

j  =  i-1  (3€) 

But  the  equation  for  the  i-th  segment  of  the  cost  curve  is  given 
by  (1)  as  C(t)  =  +  Bi(t  -  T^).  Equating  this  to  (3)  and  then 

equating  the  coefficients  of  t  gives: 

Bi  =  r  (37) 

and  then  equating  the  constant  terms  gives : 

Ai  =  (I/L  +  F)j/P  +  rTi  .  (38) 


In  this  way  the  familiar  cost  components  translate  into  a 
piecewise  continuous  cost  curve  of  equal  steps  occurring  at  equal 
intervals  and  with  the  same  slope  for  all  segments — i.e.,  there 
is  no  economy  of  scale  incorporated. 
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It  is  desirable  to  represent  some  number  of  the  discrete 
segments  of  such  a  cost  curve,  but  to  smooth  the  rest  of  the 
curve  into  a  single  linear  segment.  The  slope  for  this  segment 
will  be 

Ai  - 

Bn  =  -  =  (I/L  +  F)  +  r  (39) 

Ti  -  Ti_i 

The  starting  value  for  this  last  segment  may  be  taken  as  the 
midpoint  between  where  the  last  segment  ends  and  where  the  next 
segment  would  have  begun  using  the  normal  step: 

An  =  An-1  +  Bn-i(Tn  -  Tn-1)  +  (I/L+F)/(2P)  .  (40) 

The  decision  to  transition  to  the  smooth  curve  approximation 
is  based  on  the  cost  curve  step  height: 

Step  height  =  (I/L+F)/P  .  (41) 

If  this  height  is  small  relative  to  that  of  other  devices,  or 
relative  to  the  slope  associated  with  variable  operating  costs, 
then  the  transition  may  be  made  after  only  one  or  two  segments, 
or  perhaps  the  curve  can  be  treated  as  simple  linear  from  the 
start.  "Big  ticket"  items  such  as  full  mission  simulators,  on 
the  other  hand,  occasion  large  steps,  so  five  or  more  segments 
may  be  desirable  before  transitioning  to  the  smooth 
approximation.  In  any  case,  the  total  number  of  combinations 
across  all  devices,  defined  as  the  product  of  the  number  of 
segments  in  the  associated  cost  curves,  must  be  kept  to  some 
manageable  level  —  less  than  100,  say  —  or  the  solution  time 
for  the  Resource  Allocation  module  will  become  unacceptable. 


simple  Linear  Cost  Function 

The  simple  linear  cost  function  employed  by  the  other  OSBATS 
modules  is  simply  that  obtained  by  applying  (7)  from  the  start, 
i.e.,  approximating  the  cost  curve  as  simple  linear  with  slope 

B  =  (I/L  +  F)  +  r  .  (42) 


Optimization  Problems  and  Techniques 

Summary 

The  general  optimization  problem  addressed  by  OSBATS  is  to 
determine,  for  each  task  to  be  trained,  the  sequence  of  training 
devices,  and  the  associated  allocation  of  student  time,  that  will 
cause  the  student  to  achieve  criterion  performance  at  minimmn 
total  cost.  (The  algorithms  developed  to  solve  this  problem  may 
also  be  used  explore  the  complementary  problem:  to  maximize 
training  achieved  for  a  given  fixed  cost.) 

An  implicit  assumption  underlying  the  OSBATS  models  and 
associated  optimization  algorithms  is  that  when  a  student 
switches  from  one  training  device  to  another  he  or  she  transfers 
all  of  his  or  her  operational  performance  score  attained  to  date. 
Without  this  assumption  the  optimization  problem  becomes  much 
more  complex . 

This  general  optimization  problem  may  be  cast  as  a  sequence 
of  increasingly  more  complex,  difficult,  and  realistic  problems, 
graded  both  by  the  number  of  tasks  considered,  the  complexity  of 
the  cost  function,  and  the  existence  of  allocation  or  performance 
constraints .  OSBATS  employs  the  simplified  problems  and 
associated  algorithms  in  the  earlier  design  modules.  The  more 
complex  problems  are  tackled  in  the  Resource  Allocation  module. 
From  easiest  to  most  difficult,  the  hierarchy  of  optimization 
problems  actually  or  potentially  underlying  OSBATS  is  as  follows: 
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o  Single-task  problem  with  simple  linear  cost  function, 
and  with  no  allocation  or  entry  level  constraints . 

o  Single-task  problem  with  piecewise  linear  cost 
function,  and  with  allocation  and  entry  level 
constraints . 

o  Single-task  problem  with  continuously  nonlinear  cost 
function,  and  with  allocation  and  entry  level 
constraints . 

o  Multiple-task  problem  with  simple  linear  cost  function, 
and  with  no  allocation  or  entry  level  constraints . 

o  Multiple-task  problem  with  piecewise  linear  cost 

function,  plus  allocation  and  entry  level  constraints. 

Three  optimization  approaches  have  been  considered  in 
tackling  these  problems.  The  first  is  an  analytic  technique 
based  on  necessary  conditions  found  from  calculus,  it  is  very 
fast  and  generally  optimal  for  the  first  class  of  problem,  though 
global  optimality  is  not  guaranteed.  The  second  technique  is 
dynamic  programming,  which  is  optimal  for  any  of  the  single-task 
problems,  but  which  is  much  more  time  consiiming  than  the  analytic 
method.  The  third  technique  is  to  employ  heuristic  procedures, 
coupled  with  the  analytic  optimization  technique.  This  approach 
should  provide  good  solutions  in  most  cases. 

Although  dyneunic  programming  has  been  used  in  past  model 
development,  and  in  checking  heuristic  approaches  to  the  more 
complex  single-task  problems,  it  is  not  employed  in  the  delivered 
OSBATS  model.  Therefore,  it  will  not  be  described  at  this  time. 

The  following  exposition  will  focus  on  the  simplest  problem 
and  the  analytic  approach  to  its  solution,  and  will  then  indicate 
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the  modifications  and  heuristics  employed  to  adapt  this  analytic 
model  to  the  more  complex  problems. 

Single-Task  Optimization  Problem  with  Simple  Linear  Cost  Function 

This  problem,  which  lies  at  the  heart  of  the  Training  Device 
Selection  module,  is  to  determine  the  sequence  of  training 
devices,  and  the  associated  allocation  of  student  time,  that  will 
cause  the  student  to  achieve  criterion  performance  at  minimum 
total  cost,  if  only  one  task  is  considered  for  training.  No 
constraints  (other  than  nonnegativity)  are  placed  on  allocations 
of  student  time  or  on  entry  performance  to  a  task. 

Algorithm  Description.  The  general  strategy  of  the  analytic 
aigoritnm  is  to  move  from  one  incumbent  training  device  to  the 
next,  beginning  with  that  device  that  gives  the  greatest 
operational^  initial  performance  increase  per  dollar.  A  switch 
from  the  inciambent  device  to  a  successor  is  made  when  the 
successor's  rate  of  increase  per  dollar  catches  up  to,  and 
prepares  to  pass,  that  of  the  current  incumbent  device.  Each 
crossover  point  is  determined  by  finding  the  performance  level  at 
which  the  derivative  of  performance  with  respect  to  per-student 
dollar  costs  for  the  incumbent  training  device  is  matched  by  that 
of  a  challenger  device.  At  this  point  the  challenger  becomes  the 
incumbent  and  the  incumbent  is  removed  from  further 
consideration . 

The  algorithm  may  be  implemented  either  with  time  running 
forward  or  time  running  backward.  For  this  problem,  with  simple 
linear  cost  functions  and  no  entry  or  resource  constraints,  the 
solution  will  be  the  same  for  either  implementation. 


^The  term  "operational"  performance  is  used  to  indicate  that 
the  this  is  the  performance  the  student  would  score  if  tested  on 
actual  equipment.  It  is  the  P{t)  defined  at  the  beginning  of 
this  appendix,  and  includes  the  transfer  ratio  also  defined 
earlier. 
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Forward  Time  Implementation.  If  time  is  to  run  forward,  the 
specific  steps  are  detailed  as  follow; 

Step  0;  Let  =  Pg/  i.e.,  set  the  current  performance 

level  equal  to  expected  student  entry  level  for  the  task. 

Step  1 t  Choose  as  the  initial  training  device  that  device 
i  having  the  greatest  initial  rate  of  perfomance  increase  with 
respect  to  per-student  dollar  expenditures,  D.  That  is,  choose 
the  device  i  for  which 


dD 


i/Y 


^i  ^  ”*i‘  ^ 
~ 


k  +  1 


(1  -  y/a^) 


is  greatest.  The  selected  device  becomes  the  incumbent. 

Step  2 ;  For  each  "surviving"  training  device  j,  determine 

dy 

the  crossover  performance  level  y .  at  which  — 

J  dD 


i.y 


dD 


D/Y 


i.e.,  at  which  the  performance  improvement  rate  for  device  j 
matches  that  of  the  incumbent.  This  crossover  performance  level 
is  calculated  from  the  following  formula^: 


= 


aj  -  «i^ 


1  -  z 


where 


.  ii 


fli 

“j  '’i 


r 

r+1 


^An  equivalent  formula  is 

yj  =  aiaj  (1  -  z)/(aj  -  aiz) 

where 

z  =  [ (aj/ai) (mj/mi) (hi/hj) 
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j  ^now' 


remove 


Step  2a;  yj  not  exist,  or  y 

training  device  j  from  further  consideration. 


Step  2b;  If  no  candidate  training  device  survives,  go 
to  Step  4  to  terminate. 


Step  3 ;  Let  the  next  incumbent  be  that  training  device  j 

for  which  y  <  yj  <  Y  (the  task  training  standard)  and  for 

now  J 

which  y^  <  yj^  for  every  other  surviving  training  device  k. 
Determine  time  t^^  required  on  the  incumbent  training  device 


as 


^i  = 


m.  c 
1 


,-1/r 


(1  -  -  (1  -  y„„„/a.) 


-1/r 


The  cost  of  training  on  the  incumbent  device  is  just  h.t.. 

Let  j  become  the  new  incumbent  training  device  and  set  the 
current  performance  level  at  the  crossover  performance,  i.e.,  set 
i  -  j  and  •  y^  . 

Then  go  to  Step  2. 

Step  3a;  If  no  such  training  device  j  exists,  then  go 
to  Step  4  to  terminate. 

Step  4;  Train  on  the  incumbent  training  device  until  the 
training  standard  Y  is  achieved.  Allocate  training  time 


(1  -  Y  /a.)"^^^  «  (1  _  y 
'  1'  '  -^^now  1' 

for  this  purpose.  The  per-student  cost  of  training  on  this 
device  is 

Backward  Time  Implementation.  The  problem  may  be  solved  in 
backward  time,  deallocating  training  resources  until  performance 
is  brought  from  the  training  standard  down  to  the  entry 
performance  level.  For  this  essentially  unconstrained 
optimization  problem  the  solution  will  be  identical  to  that 
obtained  in  the  forward  time  implementation.  Such  a  backward 
time  implementation  may  seem  unnatural,  but  is  a  device  used  in 
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dynamic  programming,  a  mathematical  optimization  technique.  It 
is  found  useful  in  OSBATS  when  solving  resource  allocation 
problems  with  allocation  and  performance  entry  constraints,  which 
will  be  discussed  later. 

Algorithm  Derivation  and  Justification.  The  various 
formulas  used  in  the  algorithm  will  now  be  justified.  To  begin, 
we  restate  the  basic  form  of  the  i-th  learning  curve: 


y  =  P^(t)  =  a^  [”l  -  [1  +  m^c  (S^  +  t)]"^^ 

where  S.  is  the  head  start  used  to  adjust  Pj^(O)  to  a  desired 
beginning,  or  reference,  performance  level.  The  inverse  learning 
curve  is  given  by 

t  =  Pi^(y)  =  —  [Ti  -  y/a 

L 


If  device  i  is  used  to  train  between  two  performance  levels,  then 
the  time  to  go  from  the  one  performance  level,  y^^,  to  another 
performance  level,  Y2>  found  as  the  difference  ”  ^2  ”  ^1’ 


m^c  L__ 


(1 


yj/ai) 


This  formula  will  be  seen  twice  in  the  algorithm  formulas,  above. 
This  is  a  particularly  convenient  formula  in  that  the  head  start 
S.  has  cancelled  out.  By  focusing  the  algorithm  on  performance 
levels  rather  than  times,  this  complicating  variable  is  avoided. 


The  transition  from  one  device  to  another  occurs  when  the 
derivatives  of  performance  with  respect  to  dollars  are  equal. 
The  reasoning  for  this  is  simple.  We  want  to  increase 
performance  as  much  as  possible  for  any  expenditure  of  student 
time  (and  the  associated  costs).  Since  we  may  switch  from  one 
training  device  to  another  at  any  time  without  penalty,  we  will 
simply  want  to  "ride"  that  device  giving  the  highest  rate  of 
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improvement.  Since  performance  is  a  continuous  function  of 
per-student  attributed  costs  for  all  devices,  the  rate  of 
improvement  on  one  device  can  only  overtake  the  rate  of 
improvement  on  another  device  by  passing  through  a  performance 
level  at  which  the  rates  of  improvement  on  the  two  devices  are 
equal.  To  determine  this  crossover  performance  level,  equate 
the  expressions  for  the  rates  of  improvement  on  the  two  devices, 
and  solve  for  the  performance  level.  First,  we  need  to  express 
performance  on  device  i  as  a  function  of  per-student  costs  D^. 
Since  where  t^^  is  per-student  time  spent  on  device  i, 

we  substitute  t^  =  learning  curve  formula  to  give, 

after  some  simple  algebraic  manipulation: 


r  ""i*^  -rH 

D^)  =  a.  1  -  [1  +  {S.  +  D.  )]  ^  . 


y  { 


^i 


(It  is  useful  to  note,  at  this  point,  that  this  may  be  solved  for 
the  second  bracketed  term  to  give 

-1/r 


m^^c 


[1  +  (S.  +  D.  )]  =  (1  -  y/a.) 


a  relation  that  will  prove  useful  shortly. ) 


Now  differentiate  with  respect  to  to  give 


dy 


m.c  m.c  -r-1 

a.r  [1  +  ^  (S^  +  D.  )] 

hi  hi 


Combining  the  last  two  equations  permits  us  to  simplify  to 


dy  m.c  1+1/r 

-  »  a.r  -  [1  -  y/a.] 

dD,  h,  ^ 


A  similar  expression  holds  for  another  device  j.  The  crossover 
performance  between  the  two  devices  is  found  by  setting 
dy  dy 

dD^  dDj 
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or 


m.c  1+1/r  m.c  1+1/r 

a^r  [1  -  y/a^]  =  a^r  J —  [1  -  y/a^] 

hi  hj 

which  may  be  solved  for  y  to  yield 


Yj  =  a.a^ 


-  ^i^ 
1  -  z 


where 


fi  ill 

“j  “j  ''i 


r 

r+1 


the  expression  given  in  the  algorithm. 


Special  Cases.  Note  that  the  crossover  solution,  given  by 
the  above  equations,  is  indeterminate  if  z  =  1.  This  will  occur 
if  device  i  and  j  have  identical  cost  rates  and  learning  curves. 
It  will  also  occur  if  m^/hi  =  mj/hj  and  a^  =  aj;  in  this  case 
the  lesser  training  rate  of  the  one  device  is  just  compensated  by 
its  lesser  cost  rate.  It  is  conceivable,  though  unlikely,  that 
two  devices  that  are  indistinguishable,  in  this  sense,  might  be 
generated  in  OSBATS.  The  current  version  of  OSBATS  makes  no 
provision  to  properly  handle  the  transition  between  such  devices. 
Since  the  optimization  logic  to  handle  this  situation  would  be 
rather  complex,  it  is  suggested  that  a  simpler  expedient  be 
employed  in  future  OSBATS  work,  should  this  be  deemed  a  problem. 
The  idea  is  to  (1)  test  device  parameters  to  determine  if  any  two 
devices  are  identical,  and,  if  so,  to  (2)  randomly  perturb  the 
parameters  for  one  device  by  a  small,  inconsequential  amount,  say 
by  multiplicative  factors  drawn  from  a  normal  distribution  with 
mean  1  and  variance  0.00001.  This  tie-breaking  procedure  would 
be  sufficient  to  allow  use  of  the  optimization  algorithms 
described  here  without  more  complex  logic. 


Another  situation  of  theoretical  concern  is  that  of  multiple 
points  of  identical  performance  per  dollar  for  two  devices .  That 
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cannot  occur  with  the  restricted  family  of  learning  curves 
employed  in  OSBATS. 


Single-Task  Optimization  Problem  with  Piecewise  Linear  Cost 
Functions,  plus  Allocation  and  Device  Entry  Performance 
Constraints 


This  problem  may  be  stated  as  follows:  Find  the  training 
times  {t^}  for  a  prescribed  sequence  ot  training  devices  {i}  so 
that  the  student  is  trained  from  the  task  entry  performance  level 
Pg  to  the  standard  Y  at  minimiim  total  training  cost  C,  subject  to 
minimum  entry  performance  constraints  {y^^  >  and  maximum 

usage  constraints  {t^  <  Uj^}. 

This  problem  has  been  solved  using  dynamic  programming,  but 
solution  time  was  such  as  to  preclude  interactive  employment  in 
OSBATS.  Therefore,  an  analytic-based  heuristic  algorithm  has 
been  developed.  This  algorithm  looks  at  every  possible 
combination  of  cost  curve  segments  across  the  different  devices 
and  solves  the  optimization  subproblem  with  training  time 
allocations  to  the  various  devices  restricted  to  the  respective 
cost  segments.  The  first  combination  considered  will  have  usage 
of  each  device  restricted  to  a  particular  segment  of  the 
respective  cost  curve.  The  second  combination  will  have  device 
usage  restricted  to  other  segments.  Every  possible  combination 
is  considered  in  this  way,  in  principle.  However,  a  branch-and- 
bound  technique  is  used  to  preclude  exhaustive  examination  of 
every  combination,  in  some  cases.  The  technique  associates  with 
each  combination  a  minimum  possible  cost,  namely  the  total  cost 
if  each  device  is  used  only  as  much  as  the  value  of  the  lower 
ordinate  for  the  associated  cost  segment  for  that  combination. 

The  actual  feasible  cost  to  train  for  this  combination  will  be  at 
least  as  great  as  this  lower  bound  value.  Thus  if  at  any  point  a 
solution  has  already  been  found  whose  cost  is  less  than  the  lower 
bound  cost  for  a  given  combination,  that  combination  can  be  ruled 
out  of  consideration.  By  ordering  the  combinations  according  to 
their  lower  bound  costs,  it  is  possible  to  rule  out  all  remaining 

B-42 


cost  curve  combinations,  once  one  is  found  whose  lower  bound  cost 
exceeds  the  incvmibent  solution  cost.  This  technique  is  employed 
in  OSBATS.  However,  most  combinations  usually  must  be  examined 
in  detail,  so  the  number  of  segments  used  in  defining  each  device 
cost  curve  must  be  kept  small,  else  the  total  number  of 
combinations  will  become  too  great  to  be  considered  in  reasonable 
time. 


Strictly  defined,  each  subproblem  would  have  the  added 
constraints  that  ^  ^  "^ik^  where  tTj^  and  are  the  lower 

and  upper  bounds  on  student  time  for  combination  k  and  device  i. 
However,  we  may  ignore  the  lower  bound  if  the  following 
assumptions  are  valid: 

Asstimptions :  (1)  No  cost  curve  has  a  negative  step  between 

a  cost  curve  segment  and  the  succeeding  segment.  (2)  No  cost 
curve  has  a  segment  whose  slope  is  greater  than  a  preceding 
sloped . 

For  suppose  these  assiunptions  are  true,  and  suppose  the 
subproblem  algorithm  finds  a  solution  violating  the  lower  bound 
for  some  cost  curve  segment  under  a  given  combination.  Then  the 
cost  attributed  to  that  device  will  exceed  the  true  cost,  since 
(1)  the  backward  projection  of  the  current  segment  necessarily 
lies  above  the  preceding  segments;  therefore,  some  other 
combination,  for  which  the  appropriate  cost  curve  segment  is 
prescribed,  will  yield  the  same  training  effectiveness  ot  lower 
cost.  Since  all  combinations  are  considered,  the  lowest  cost 
will  be  attributed  to  some  combination,  and  for  that  combination 


®The  first  of  these  assximptions  is  quite  reasonable  in  any 
serious  costing  situation,  since  the  converse,  e.g.,  "5  apples 
for  one  dollar  or  6  apples  for  80  cents"  would  seem  to  have  no 
rationale  other  than  a  bizarre  promotional  giiranick.  The  second 
assiimption  is  tantamount  to  asstiming  that  there  are  no 
diseconomies  of  scale,  e.g.,  that  devices  are  not  being  drawn 
from  a  limited  pool  (such  as  real  estate  land)  causing  the  price 
to  rise  with  demand.  We  expect  that  economies  of  scale  —  rather 
than  diseconomies  —  to  be  operative  in  training  device  costing, 
so  believe  the  assumption  valid. 
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the  lower  bounds  will  be  respected.  Therefore,  we  may  ignore  the 
lower  bound  and  simply  impose  a  single  upper  bound  constraint: 

t.  <  u.  =  minimum  (T^,  ,  U.  )  . 

Suppose  a  particular  cost  curve  combination  has  been 
selected.  We  transform  the  associated  subproblem  to  one 
involving  a  simple  linear  cost  curve  without  setup  cost  by 
setting  aside  the  setup  cost  for  each  device,  solving  the 
subproblem  using  the  simple  linear  cost  rate  appropriate  to  the 
assumed  cost  curve  segment  for  each  device,  and  then  adding  in 
the  setup  cost  after  the  solution  to  the  simplified  subproblem 
has  been  found. 

The  analytic  algorithm  developed  for  solving  single  task 
problems  with  simple  linear  cost  functions,  discussed  earlier, 
cannot  be  used,  without  modification,  to  solve  the  current 
constrained  problem.  The  problem  is  that  the  "natural"  points  of 
economic  transition  from  one  device  to  another  may  correspond  to 
violation  of  resource  constraints  or  entry  performance 
constraints.  Thus,  hexiristic  rules  have  been  incorporated  to 
prevent  such  violations .  These  rules  also  have  been  formulated 
to  enforce  the  following: 

Assumption:  No  training  device  should  be  used  more  than 
once  in  the  training  of  a  particular  task.  This  will  be  true 
automatically  for  the  unconstrained  problem,  given  the  assumed 
family  of  learning  and  cost  curves. 

The  revised  algorithm  incorporates  the  following  rules:  (1) 
choose  the  best  allowable  device  for  training  the  task  at  the 
current  performance  level  (where  "allowable"  indicates  that 
neither  performance  entry  nor  resource  usage  constraints  are 
violated),  (2)  if  possible,  use  the  "incumbent"  device  until  a 
"natural"  transition  to  another  device  is  reached,  based  on 
training  benefit/cost  consideration  (i.e.,  the  crossover 
performance  levels  discussed  earlier),  but  (3)  if  that  "natural" 
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performance  transition  cannot  be  reached  without  violating 
constraints,  then  allocate  only  sufficient  resources  to  the 
"incumbent"  to  bring  the  performance  to  that  constraint  boundary, 
then  switch  to  the  most  efficient  allowed  device  at  that 
performance  level. 

Following  these  rules  may  result  in  a  device  being  used  more 
than  once  in  the  training  sequence.  To  prevent  this  undesirable 
situation  from  persisting,  the  total  amount  of  resources 
allocated  to  the  device  is  remembered  and  then,  on  a  new,  fresh 
allocation  pass,  this  amount  of  resources  is  forced  to  be  used 
all  at  the  same  time. 

One  further  modification  is  to  use  the  backward-in-time 
allocation  approach  mentioned  earlier.  This  was  found  to  prevent 
the  allocation  of  large  amounts  of  student  time  to  "entry  level" 
devices  in  an  attempt  to  make  up  for  resource  shortfalls  for  some 
other  devices. 

This  completes  the  basic  ideas  behind  the  revised  single¬ 
task  algorithm.  The  details  of  the  algorithm,  presented  in  the 
body  of  the  report,  should  thus  be  illuminated. 


Single-Task  Problem  with  Continuously  Nonlinear  Cost  Function, 
and  with  Allocation  and  Entry  Level  Constraints. 

This  problem  is  mentioned  just  for  the  record.  It  can  be 
solved  using  dynamic  programming  if  the  training  sequence  is 
specified.  This  was  done  in  earlier  OSBATS  research.  It  was 
abandoned  for  several  reasons;  (1)  run  times  promised  to  be  much 
longer  than  with  approaches  based  on  tht  analytic  determination 
of  crossover  performance  levels;  (2)  it  was  preferrable  to  not 
pre-specify  the  training  sequence;  and  (3)  it  was  decided  that 
the  generality  provided  by  piecewise  linearity  was  adequate  for 
cost  curve  representation. 
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Multiple-Task  Problem  with  Simple  Linear  Cost  Function,  and  with 
No  Allocation  or  Entry  Level  Constraints 

This  problem,  too,  is  included  simply  for  the  record.  It 
may  be  solved  by  simply  solving  the  constituent  single-task 
problems  and  aggregating  the  allocations . 


Multiple-Task  Problem  with  Piecewise  Linear  Cost  Function,  Plus 
Allocation  and  Entry  Level  Constraints 

This  is  the  problem  actually  addressed  by  OSBATS.  The 
algorithmic  procedure  is  described  in  detail  in  the  body  of  the 
report,  and  will  not  be  repeated  here.  The  algorithm  will  be 
briefly  discussed  and  related  to  the  single-task,  constrained 
problem  discussed  in  some  depth  above,  and  upon  which  it  is 
built.  As  with  the  single-task  algorithm,  the  multiple-task 
algorithm  examines  each  of  the  mathematically,  possible 
combinations  of  cost  curve  segments  to  see  if  it  is  possible  to 
satisfy  the  training  requirements  under  that  cost  structure,  and 
to  determine  if  the  total  training  cost  is  better  than  offered  by 
other  combinations . 

The  multiple-task  problem  is  complicated  by  the  fact  that 
device  usage  constraints  (user-defined  or  cost-segment  bounds) 
run  across  all  tasks.  Thus,  though  the  single-task  algorithm  is 
capable  of  solving  a  single-task  problem  while  respecting  device 
usage  constraints,  the  aggregate  of  the  single-task  solutions 
across  tasks  may  very  likely  violate  these  constraints.  We  have 
not  been  able  to  discover  an  optimal  algorithm  to  solve  this 
problem  —  with  or  without  dynamic  programming.  So  we  have 
worked  to  develop  satisfactory  heuristics. 

The  idea  lying  behind  the  current  heuristic  algorithm  is 
that  if  a  particular  device  proves  so  attractive,  from  a 
training-per-cost  standpoint,  that  unconstrained  optimization 
calls  for  it  to  be  used  more  than  allowed  by  the  constraints, 


then  the  device  should  be  made  less  attractive  by  raising  its 
price,  and  the  unconstrained  problem  solved  again.  By  adjusting 
device  cost  rates  using  surcharge  variables,  an  iterative 
procedure  should  eventually  bring  the  problem  to  solution,  if  it 
is  feasible.  If  the  problem  is  infeasible,  the  surcharge 
variables  will  tend  to  flip-flop,  and  no  convergence  will  be 
obtained.  The  idea  of  surcharges  comes  from  a  class  of  nonlinear 
programming  approaches  falling  under  the  category  of  "penalty" 
methods,  or  "SUMT  (sequential  unconstrained  minimization 
technique ) " . 

In  setting  surcharges,  it  is  not  sufficient  simply  to  drive 
the  usage  of  all  devices  below  the  constraint  limits.  Upon 
algorithm  termination,  surcharges  greater  than  1  should 
correspond  to  device  usages  driven  down  to,  but  not  below, 
allowed  usage.  No  surcharges  should  be  (significantly)  less  than 
1  at  termination.  (These  conditions  on  the  surcharges  are 
reminiscent  of  the  Kuhn-Tucker  conditions  of  constrained 
optimization  programming.)  Thus,  the  algorithm  is  designed  to 
adjust  surcharges  until  these  conditions  are  true,  within  certain 
tolerances . 


i 

I 
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GLOSSARY 


Variables  and  Parameters  Used  in  Media  Effectiveness  Discussion'^ 

a^  =  Transfer  ratio  for  transforming  performance  on  a 

training  device  with  fidelity  vector  r  to  performance 
on  actual  equipment.  [ASM^i] 

aR  =  Benchmark  asymptote  for  task  T  on  hypothetical  device 
R.  [ASMtr]. 

bi  =  Parameter,  associated  with  fidelity  dimension  i,  used 
in  fitting  transfer  ratio  equation,  [b^j] 

c  =  c;^  =  Learning  curve  time  scaling  factor  for  actual 
equipment.  [TSF^j 

c^i  =  Learning  curve  time  coefficient  for  training  device. 

[  =  TMti  TSFt] 

f()  =  General  notation  for  mathematical  function.  [n.a.] 

fi  =  Fractional  degradation  in  transfer  that  would  occur  as 
the  fidelity  on  the  i-th  fidelity  dimension  is 
decreased  from  1  to  0,  with  all  other  fidelity  levels 
held  at  1.  [FMINj,] 

i  =  Running  index  identifying  fidelity  dimension,  [j] 

Id  =  Index  of  training  device  instructional  features 
benefit.  [IADJ<ri] 

k  =  Learning  curve  shape  pareuneter.  [r] 

m  =  Time  multiplier  used  to  account  for  increased  training 
efficiency  due  to  instructional  features  and  reduced 
setup  time.  [TMipj_] 

P(t)  ==  Performance  on  actual  equipment  for  the  task  under 
consideration  after  training  on  the  device  under 
consideration  for  an  cunount  of  time  t.  [PTi(^)  when 
i  corresponds  to  actual  equipment] 


Pd(t)  =  Performance  score  on  the  training  device  in  question 
after  training  a  time  t  on  the  device;  this  is 
scored  on  a  0-1  scale  and  represents  the  score  on 
the  training  device  itself,  and  does  not  include  any 
estimate  of  transfer  to  actual  equipment. 

[  =  P(t)/ASMTi] 


^Long-form  names,  as  used  in  the  body  of  this  report,  are 
identified  in  brackets  at  the  end  of  each  entry,  when  applicable. 
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Pg  =  student  entry  performance  for  the  task  in  question. 
[PENTRY,  ENT-r] 

q  =  Parameter  (multiplicative  coefficient)  used  to  fit 
transfer  ratio  formula.  [Qt] 

Q  =  Parameter  (exponent)  used  to  fit  transfer  ratio  formula. 

[QtI 

r^  =  Fidelity  level  characteristic  of  device  in  question  on 
i-th  fidelity  dimension,  as  measured  on  0-1  scale. 
[FCAPij] 

Ri  =  Target,  or  required,  fidelity  level  on  i-th  fidelity 

dimension  in  order  to  train  task  in  question.  [FRQTrpj] 

s  =  Effective  head  start,  chosen  so  that  P(0)  or  Pd{0) 

equals  the  entry  performance  for  the  device  in  question. 
[HSti] 

sj^  =  Head  start  for  actual  equipment.  [HStjl  when  i 
corresponds  to  actual  equipment] 

Sd  =  Head  start  for  training  device.  [HS«ri  when  i 

corresponds  to  device  other  than  actual  equipment] 

t  =  Time  allocated  for  each  student  on  the  device  in 
question  for  training  the  task  in  question.  [t] 

ts  =  Time  required  to  achieve  task  performance  standard  by 
training  on  actual  equipment  only.  [TTOT-p] 

uj^  =  Ratio  of  useful  to  total  training  time  on  actual 
equipment.  [Ut,AE1 

Ud  =  Ratio  of  useful  to  total  time  on  training  device. 

[UTi] 

X  =  Parameter  used  in  defining  meaning  of  {Ri}  in  terms  of 
Y.  [SADJ] 

y  =  Performance  level,  used  as  a  variable.  [n.a.] 

Y  =  Criterion,  or  standard,  performance  level  for  task  in 
question.  [STD<r] 

2  =  Utility  variable.  Used  as  dummy  variable  in  defining 
specific  functional  form  f(z).  Also  used  as 
intermediate  variable  in  equation  for  Q.  [z] 

Variables  and  Parameters  Used  in  Cost  Function  Discussion 

Ai  -  Height  of  cost  curve  C{t)  at  beginning  of  i-th  segment, 

t  Ain  ] 
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Bi  -  Slope  of  i-th  segment  of  cost  curve  C(t).  [Bin] 

C(t)  =  Training  device  costs  attributed  to  each  individual 
student  if  each  student  uses  training  device  for 
amount  of  time  t.  [CUMCi(t)] 

F  -  Fixed  operations  cost,  annual.  [FOCi] 

H  -  Maximum  potential  annual  usage  (student  hours)  of  device 
per  yea;  .  considering  school  operating  hours,  device 
maintenance  schedules,  number  of  students  served 
simultaneously,  etc.  [UMAXi] 

i  -  Index  of  segment  (interval)  of  cost  curve.  [n] 

I  -  Investment  cost  for  each  training  device.  [INVi] 

j  -  Total  niimber  of  devices  acquired  (for  given  t).  [n] 

L  -  Life  cycle  for  each  training  device.  [LCi] 

N  -  Nvunber  of  segments  in  the  cost  curve.  [N] 

P  -  Number  of  students  trained  annually  in  the  training 
program.  [GRADS] 

r  -  Variable  operations  cost,  per  student  hour.  [VHRCi] 

t  =  Expected  total  time  that  an  individual  student  will  use 
the  training  device  in  question  during  his  training 
program.  [t] 

Ti  -  Starting  value  for  i-th  segment  of  cost  curve  C(t). 
[Tin] 

Variables  Used  in  Discussion  Optimization  Problems  and  Techniques 

ai  =  Asymptote  for  training  device  i.  (Same  as  aj-, 
earlier.)  [ASMip^] 

c  =  Time  scaling  factor  in  learning  curves.  [TSF-j] 

D  =  "Dollars",  cumulative  per  student  training  costs,  [n.a.] 

e^  =  Minimum  performance  at  which  training  device  i  may  be 
use  to  train  task  in  question.  [EPMINTi] 

hi  =  Hourly  cost  rate  for  training  device  i.  [B^n  where 
n  represents  current  cost  curve  segment  for  device  i; 
or  VHRCi] 

i  =  Index  of  training  device,  usually  the  "incumbent"  best 
device  at  current  stage  of  allocation/assigninent .  [i] 
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j  =  Index  of  other,  "candidate"  training  devices.  [j] 

k  =  Learning  curve  shape  parameter.  [r] 

mi  =  Time  multiplier.  [TM^i] 

Pg  =  Entry  performance  level.  [ENTip] 

Si  =  Head  start  for  training  device  i.  (Same  as  S^/ 
earlier.)  [HSTi] 

t  =  Time  each  individual  student  spends  in  training  on  a 
particular  device,  [t] 

Ti“  =  Minimum  allocation  that  may  be  made  to  device  i  for 
training  current  task  under  current  cost  curve 
constraints,  [n.a.] 

Ti"*"  =  Maximum  allocation  that  may  be  made  to  device  i  for 
training  current  task  under  current  cost  curve 
constraints.  [n.a.] 

ui  =  Effective  upper  bound  on  allocation  to  device  i  for 

current  subproblem,  considering  both  user-defined,  cost 
curve  segment-defined,  or  algorithm-imposed 
constraints.  [UBTi] 

Ui  =  User-defined  upper  bound  on  allocation  to  device  i. 
[USMAXi] 

y  =  Performance  level,  in  general.  [n.a.] 


yj  =  Crossover  performance  level,  at  which  dy/dD  for  TDj 

just  equals  that  of  incumbent  TDi  (at  same  performance 
level).  [PXij  or  PXgi] 

Ynow  =  Current  performance  level.  [PnowJ 

Y  =  Training  standard  for  particular  task.  [STDt] 
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EXECUTIVE  SUMMARY 


This  paper  traces  the  history  of  overseas  deployment 
training  of  the  Artny  and  Air  National  Guard  and  the  intervention 
of  a  few  governors  in  it  beginning  in  1985. 

The  furor  after  Governor  Joseph  Brennan  of  Maine  canceled  a 
deployment  of  two  of  his  Army  Guard  units  to  Central  America  in 
1986  prompted  Congress  to  enact  what  has  become  known  as  the 
Montgomery  Amendment,  named  after  its  author,  U.S.  Representative 
G.V.  ''Sonny''  Montgomery  (D-Mississippi ) .  The  amendment,  which 
was  added  to  the  FY87  Department  of  Defense  Authorization  Act, 
withdraws  from  governors  authority  to  withhold  overseas 
deployments  for  their  National  Guard  units  on  account  of 
location,  purpose,  type  or  schedule  of  such  training.  A  lawsuit 
brought  by  the  governor  of  Minnesota  setks  to  have  that  language 
ruled  unconstitutional  as  a  violation  of  the  Militia  Clause  to 
the  U.S.  Constitution. 

The  governors'  actions  had  a  major  impact  at  the  Department 
of  Defense.  They  threatened  the  resources  of  the  National  Guard 
as  it  has  evolved  as  a  part  of  the  Total  Force.  There  was  tai.’-  of 
withdrawing  force  structure  from  the  Guard.  In  the  end,  the  chief 
of  the  National  Guard  Bureau  moved  against  the  governor  of  Ohio, 
threatening  to  ''withdraw  the  Ohio  National  Guard  from  Ohio''  if 
Governor  Richard  Celeste  didn't  consent  to  the  deployment  of  the 
16th  Engineer  Brigade  to  Honduras  in  1989.  Faced  with  a  $256 
million-a-year  bill,  he  did  and  it  did. 
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